DEV Community

Cover image for How To Configure Dynamic Build Agents In Jenkins Using AWS And Terraform
Kelvin Onuchukwu
Kelvin Onuchukwu

Posted on

How To Configure Dynamic Build Agents In Jenkins Using AWS And Terraform

Cloud computing has come of age.
According to CRN Magazine, Worldwide enterprise cloud infrastructure services spending increased by $10.5 billion year over year in third quarter 2023, representing an 18 percent growth rate. This marks the fifth successive quarter in which the cloud market grew by between $10 billion to $11 billion from the prior year.

Companies can now build out their infrastructure entirely in the cloud, eliminating the need to have any single on-prem infrastructure. For companies that implementig Continuous Integration and deployment (CI/CD) using Jenkins, a major concern is how to configure their jenkins build agents.

Jenkins build agents are a very important component of any "self-respecting" Continuous integration infrastructure built on jenkins. They are important because it is not practical, much less advisable to build on the Jenkins Master.
When you have multiple long-running builds per day, you would want to configure your Jenkins infrastructure for scalability and high availability.

There are several Jenkins build agent types available - inlcuding physical machines, virtual machines, Kubernetes clusters, and with Docker images.

In this project, we will be using AWS EC2 instances to set up build agents. We have to configure it in such a way that our set up is built for high availability, dynamic scalability and idempotency. We will utilize terraform for deploying the Jenkins Controller (master). Also we will use terraform to provision cloud resources for the build agents.

Again, here's our reference diagram:

Image description

We will not be using ansible or packer in our own case however. We will create and provision the controller and other cloud resoruces using just Terraform. Also, we won't be creating an application load balancer, as it is not needed here.

Our application will consist of a single Jenkins controller and multiple jenkins agents spread across three availability zones. The jenkins agents will be dynamically provisioned and will scale up or down depending on the average CPU utilization of the instances.
Here's the terraform code for the policy beign used:

resource "aws_autoscaling_policy" "asg-policy" {
  autoscaling_group_name =
  name                   = "jenkins-agents-asg-policy"
  policy_type            = "TargetTrackingScaling"
  target_tracking_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ASGAverageCPUUtilization"
    target_value = 50.0

  depends_on = [
Enter fullscreen mode Exit fullscreen mode

As you can see, we are using a target tracking policy which tracks the average CPU utilization and will scale up once the meric exceeds 50%.
An Elastic Filesystem will also be provisioned for persistent file storage across all the jenkins servers, should the need arise.

Step 1: Clone The Git Repository

git clone
Enter fullscreen mode Exit fullscreen mode
  • Switch into the terraform directory
cd terra-jenk/terraform
Enter fullscreen mode Exit fullscreen mode
  • Initialize and run the terraform code
terraform init
terraform plan
terraform apply -auto-approve
Enter fullscreen mode Exit fullscreen mode

Wait for the "Apply" to complete.

Image description

At this stage, half of the work is done! Such is the beauty of infrastructure-as-Code.

By using Terraform, we have been able to provision the Jenkins Controller, a Multi-AZ auto scaling group for the Jenkins agents, an Elastic Filesystem for scalable, persistent file storage - conected across all the servers, security groups and appropirate instance roles.

Step 2: Configure SSH Keys

In order for the cotroller to connect to the agents, it will need an SSH private key. Hence the launch template for the jenkins agents is configured to create a key-pair and attach to the agents.
However, terraform does not actually generate the key-pair. You will have to manually generate them and then feed the values to terraform.

Since ssh-key pairs are meant to be secret, we will be using an advanced method of handling terraform secrets.

  • First, login to your AWS account and go to the KMS service.
  • Create an AWS Customer Managed Key. I'll be using the symmetric key type.

Image description

Image description
NB: Make sure that your terraform user has sufficient IAM permissions to invoke this key.

  • In the repository, within the terraform directory (your working directory), generate an SSH key-pair.
Enter fullscreen mode Exit fullscreen mode
  • Make sure it is named id_rsa.
  • Use the AWS KMS encrypt command to encrypt your credentails file.
aws kms encrypt \
    --key-id <YOUR-CMK-KEY-ID> \
    --plaintext fileb:// \
    --output text \
    --query CiphertextBlob | base64 \
    --decode >
Enter fullscreen mode Exit fullscreen mode

This will encrypt our create an encrypted file called which will contain our credentials. We can now safely delete the file. The file is secure enough to be checked into version control.

  • Copy the private key (id_rsa) to your note pad or another text editor. It will be required in Step 4. After that, You can either delete the private key (id_rsa) or simply ignore it (It will not be commited into version control).

The terraform code has been configured to handle every other thing else.
You can also read more on How To Manage Your Terraform Secrets Like A Pro.

Step 3: Configure The Jenkins Controller

  • Log in to the jenkins controller
    After the resources have been provisioned by teraform and status checks have passed, log in to the jenkins server on port 8080.

  • Go to Dashboard > Manage Jenkins > Plugins > Available plugins

Image description

Image description

  • Install the EC2 Fleet plugin

Image description

  • Go to Manage Jenkins > Clouds > Add new cloud

Image description

  • Click on EC2 Fleet and assign a name to it. Click on Create.

We do not need to supply AWS Credentials since Jenkins will use the instance profile attached to our EC2 instance to connect to our AWs account.

  • Select the "us-east-1" region
  • Select "jenkins-agents-asg" under EC2 Fleet

Image description

When you click on "test connection", you should see a similar success message as below:

Image description

  • Click on Save

We are not done with this configuration, we will get back to it later.

Step 4: Configure SSH Credentials

  • Back on the Jenkins server, go to Dashboard > Manage Jenkins > Credentials

Image description

  • Click on "Add credentials", select "SSH Username with private key"
  • Fill in the boxes as required

Image description

  • Copy the contents of the private key that you created in your working directory in step 2 above.

Image description

Step 5: Set Up The Cloud

  • Go back to Clouds and select the EC2 Fleet you created earlier
  • Clck on Configure
  • Under "Launcher", select the credentials you just created. Also set "Host key verification strategy" to Non verifying strategy.
  • Check the box to connect via private IP

Image description

  • Check the box to Always Reconnect.
  • Leave the defualt label or modify as you wish.

Image description

  • The jenkins Filesystem Root is /var/lib/jenkins
  • Max idle Minutes before scaledown should be set to 5
  • Check the box for "No Delay Provision Strategy"

Image description

  • Click on "Save".

After some time, Jenkins will automatically connect to the running agents in your auto scaling group. It should look like this:

Image description
Notice the "EC2 Fleet ..." at the bottom left corner.

Step 6: Test The Setup

This final step is a validation step to confirm that everything is indeed working properly.

We will run a workload on this agent to see if it will run as expected.

  • Go to Dashboard > New item > Freestyle Project
  • Restrict where the project can be run by setting the label for the EC2 Fleet.

Image description

  • Click on "Add build step" > "Execute shell script"
  • Add the following shell script in the box

echo "Hello world"
sleep 15
echo "Goodbye world"
Enter fullscreen mode Exit fullscreen mode
  • Save and build.

Image description

As you can see, the build ran successfully on the remote agent!


EC2 instances are a great option for building jenkins agents.
The EC2 Fleet plugin enables dynamic provisioning of the instances based on your available fleet (autoscaling group).
Terraform is a great tool for achieving automated provisoning of cloud infrastructure in a replicable manner.

What terraform concept bothers you? What would you like to learn about terraform, infrastructure-as-code and DevOps in general? Let me know in the comments and I might consider writing a detailed tutorial on that.

PS: I am open to remote DevOps, Cloud or technical writing offers.

Connect with me on LinkedIn.

Top comments (0)