DEV Community

Cover image for How to set up Jenkins and a Pipeline on AWS
Sri for AWS Community Builders

Posted on

How to set up Jenkins and a Pipeline on AWS

Table of Contents

  1. Introduction
  2. Create an IAM User
  3. Create a Key pair
  4. Create a Security Group
  5. Create an EC2 instance
  6. Install and Configure Jenkins
  7. Create a Pipeline
  8. Clean Up
  9. Summary
  10. Referrals

Introduction

In this blog we are going to set up Jenkins on an EC2 instance and then set up a pipeline to copy a file from S3.
We are going to start something very basic and enhance it as we move along.

What is Jenkins?

Jenkins offers a simple way to set up a continuous integration or continuous delivery (CI/CD) environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks. While Jenkins doesn’t eliminate the need to create scripts for individual steps, it does give you a faster and more robust way to integrate your entire chain of build, test, and deployment tools than you can easily build yourself.

What is Jenkins Pipeline?

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines as code via the Pipeline domain-specific language (DSL) syntax.


Demo

Let's get started with the demo.

Step 1. Create an IAM User

  1. Navigate to IAM > Users > Click on Add users
  2. Enter User Name as JenkinsUser
  3. Select Access key - Programmatic access and Click Next: Permissions
  4. Click Attach existing policies directly and then Create policy
  5. In the Create policy window, Click on JSON and add the following JenkinsEC2Policy and save the policy as JenkinsEC2Policy
  6. Select AmazonS3ReadOnlyAccess and JenkinsEC2Policy under Attach existing policies directly
  7. Click Next:Tags, Next:Review and then Create user
  8. Download the credentials .csv file as we will need this during Jenkins configuration at Step 6.

    JenkinsEC2Policy

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1312295543082",
            "Action": [
                "ec2:DescribeSpotInstanceRequests",
                "ec2:CancelSpotInstanceRequests",
                "ec2:GetConsoleOutput",
                "ec2:RequestSpotInstances",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "ec2:CreateTags",
                "ec2:DeleteTags",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeRegions",
                "ec2:DescribeImages",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "iam:ListInstanceProfilesForRole",
                "iam:PassRole",
                "ec2:GetPasswordData"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
    }
    

IAM_JenkinsUser1

IAM_JenkinsUser1

IAM_JenkinsUser2

IAM_JenkinsUser2

IAM_JenkinsUser3

IAM_JenkinsUser3


Step 2. Create a key pair

  1. Navigate to EC2 > Key Pairs (under Network & Security).
  2. Click Create key pair.
  3. Navigate to the folder where the key pair is downloaded and run.
   chmod 400 <key_pair_name>.pem
Enter fullscreen mode Exit fullscreen mode

Jenkins_Keypair

Jenkins_Keypair


Step 3. Create a Security Group

  1. We are going to create a Security Group for SSH and Jenkins web access.
  2. Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
    • Name: JenkinsSG.
    • Add an Inbound rule to allow SSH (TCP 22) traffic from My IP.
    • Add another Inbound rule to allow Custom (TCP 8080) traffic from My IP.

Jenkins_SG

Jenkins_SG


Step 4. Create an EC2 instance

  1. Navigate to EC2 > EC2 Dashboard > Click on Launch instance.
  2. Launch an instance with the following values as shown in the screenshots.
  3. When the instance state is Running, select the instance and click on Connect and then copy the connections details.
  4. Connect to the instance.

    ssh -i "Jenkins-Keypair.pem" ec2-user@ec2-54-211-70-130.compute-1.amazonaws.com
    

Jenkins_EC1

Jenkins_EC1

Jenkins_EC2

Jenkins_EC2


Step 5. Install and Configure Jenkins


Install Jenkins
  1. Ensure that the software packages are up to date on the instance by executing the following command:

    [ec2-user ~]$ sudo yum update –y
    
  2. Add the Jenkins repo using the following command:

    [ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
    
  3. Import a key file from Jenkins-CI to enable installation from the package:

    [ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
    
    [ec2-user ~]$ sudo yum upgrade
    
  4. Install Java:

    [ec2-user ~]$ sudo amazon-linux-extras install java-openjdk11 -y
    
  5. Install Jenkins:

    [ec2-user ~]$ sudo yum install jenkins -y
    
  6. Enable Jenkins service to auto start at boot:

    [ec2-user ~]$ sudo systemctl enable jenkins
    
  7. Start Jenkins as a service:

    [ec2-user ~]$ sudo systemctl start jenkins
    
  8. You can check the status of the Jenkins service using the command:

    [ec2-user ~]$ sudo systemctl status jenkins
    
  9. Just in case we ever need to restart Jenkins during this setup/configuration.

    [ec2-user ~]$ sudo systemctl restart jenkins
    

Configure Jenkins
  1. Copy the Public IPv4 DNS of EC2 instance and paste the URL as following in the browser For Example: http://ec2-54-211-70-130.compute-1.amazonaws.com:8080
  2. Enter the initialAdminPassword from /var/lib/jenkins/secrets/initialAdminPassword and Click Continue

    [ec2-user@ip-172-31-89-157 ~]$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword
        671x0x5x3xxx46xxxxxx099x1xf0149x
    
  3. Select Install suggested plugins.

  4. Once the installation is complete, Create First Admin User will open. Enter your information, and then select Save and Continue.

  5. Click Dashboard, select Manage Jenkins, and then select Manage Plugins.

  6. Click Available plugins, Search and Select Amazon EC2 and then **Install without restart*

  7. Once the installation is complete, Navigate back to Dashboard, select Manage Jenkins, select Manage nodes and clouds and then click on Configure Clouds.

  8. Select Add a new cloud, and select Amazon EC2. A new pop up window opens with more fields.

  9. Click Add under Amazon EC2 Credentials and Select Jenkins

    • From the Jenkins Credentials Provider: Jenkins, select AWS Credentials as the Kind.
    • Enter Access Key ID, Secret Access Key from the key pair and Click Add.
  10. Scroll down to Region and select your Region.

  11. Click Add under EC2 Key Pair's Private Key and Select Jenkins.

    • From the Jenkins Credentials Provider: Jenkins, select SSH Username with private key as the Kind and set the Username to ec2-user.
    • Select Enter Directly under Private Key, then select Add.
    • Open the private key pair you created in the creating a key pair step and paste in the contents from -----BEGIN RSA PRIVATE KEY----- to -----END RSA PRIVATE KEY-----. Select Add when completed.
  12. Scroll down to Test Connection and ensure it states Success and then Click Save.

Jenkins_Configure1

Jenkins_Configure1

Jenkins_Configure2

Jenkins_Configure2

Jenkins_Configure3

Jenkins_Configure3

Jenkins_Configure4

Jenkins_Configure4

Jenkins_Configure5

Jenkins_Configure5

Jenkins_Configure6

Jenkins_Configure6

Jenkins_Configure7

Jenkins_Configure7

Jenkins_Configure8

Jenkins_Configure8

Jenkins_Configure9

Jenkins_Configure9

Jenkins_Configure10

Jenkins_Configure10

Jenkins_Configure10.1

Jenkins_Configure10.1

Jenkins_Configure10.2

Jenkins_Configure10.2

Jenkins_Configure11

Jenkins_Configure11

Jenkins_Configure11.1

Jenkins_Configure11.1

Jenkins_Configure11.2

Jenkins_Configure11.2

Jenkins_Configure12

Jenkins_Configure12


Step 6. Create a Pipeline

  1. Navigate to Dashboard > Manage Jenkins and copy the ID of IAM User created in Step 1, which we need to replace xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the ID.
  2. Navigate to Dashboard > Manage Jenkins > Plugin Manager > Click on Available plugins
  3. Search for AWS Steps and Install without restart.
  4. Navigate to Dashboard > New Job > Enter download-a-file-from-s3 > Select Pipeline and Click OK.
  5. Scroll down to Pipeline, add the following Pipeline script and Click Save
  6. Create an S3 bucket and copy some files to the bucket.
  7. Replace the s3bucket with your bucket name and filename with one of the files in your S3 bucket.
  8. Navigate to Dashboard > download-a-file-from-s3 and Click on Build Now.
  9. Navigate to the latest Build History link and check out the Console Output.

    pipeline 
    {
    agent any
        stages
        {
        stage('S3download') 
            {      
                steps {
                    withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                    {
                        s3Download(file: "filename", bucket: 's3bucket', path: '')
                    }
                }
            }
        }
    }
    

Jenkins_Pipeline1

Jenkins_Pipeline1

Jenkins_Pipeline2

Jenkins_Pipeline2

Jenkins_Pipeline3

Jenkins_Pipeline3

Jenkins_Pipeline4

Jenkins_Pipeline4

The following is the console log, which shows the pipeline has been successful. ✅

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh 
Finished: Downloading from s3bucket/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Enter fullscreen mode Exit fullscreen mode

The file that was copied from S3 to the EC2 instance. ✅

[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/
total 0
drwxr-xr-x 2 jenkins jenkins 55 Dec 31 03:45 receiveMessages.sh
Enter fullscreen mode Exit fullscreen mode

Let's run the build again, this time the build has failed with the following error ❓
The error message suggests to use set force=true.

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/ 
Download failed due to existing target file; set force=true to overwrite target file
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Target exists: file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:146)
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:113)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE
Enter fullscreen mode Exit fullscreen mode

So we need to add force:true as per the Pipeline: AWS Steps documentation

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    s3Download(file: "filename", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

The build succeeded after adding force:true and now we can run the build multiple times.

Now let's parameterise the destination foldername.
  1. Navigate to Dashboard > download-a-file-from-s3 and click on Configure
  2. Select This project is parameterised and then Add String parameter
  3. Enter Name as foldername, Default Valu as foldername and Save
  4. Click Build with Parameters
  5. Navigate to the latest Build History link and check out the Console Output.
pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    echo "${foldername}" 
                    s3Download(file: "${foldername}", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Jenkins_Pipeline_Parameter

Jenkins_Pipeline_Parameter

Jenkins_Pipeline_Build-with-Parameters

Jenkins_Pipeline_Build-with-Parameters

The following is the console log, which shows the pipeline has been successful. ✅

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] echo
s3files
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
Finished: Downloading from sqssri/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Enter fullscreen mode Exit fullscreen mode

The files were copied to s3files directory on the EC2 instance. ✅

[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
total 8
-rw-r--r-- 1 jenkins jenkins 952 Dec 31 03:40 receiveMessages.sh
-rw-r--r-- 1 jenkins jenkins 646 Dec 31 03:40 sendMessages.sh
Enter fullscreen mode Exit fullscreen mode

Clean Up

  1. Delete the EC2 instance.
  2. Delete the JenkinsUser.
  3. Delete the key pair.
  4. Delete the Security Group JenkinsSG.

Summary

  • We learned how to install and configure Jenkins.
  • We also learned about setting up the pipeline.

Referrals

Top comments (8)

Collapse
 
anusachdeva4321 profile image
Anu Sachdeva

Thank you for the descriptive article. I have a few questions please 1) Why was the S3 bucket not made public and no bucket policy attached to let Ec2 access this s3.
2) why was no role attached to the ec2 to access the S3 bucket?
I had to follow some more additional steps to complete this setup. I had to do below coz I was getting 403 access denied error. I am not sure which one step exactly worked because I feel it is cumulation of all these steps.
1) Made my s3 bucket public 2) Attached below policy to my S3 bucket . 3) Attached an IAM role to my Jenkins ec2 so that it can access s3(not sure if this was actually needed or not but I tired everything to resolve the 403 error I was getting) . After following these steps my 403 error message went away and the job build was success
{
"Version": "2012-10-17",
"Id": "Policy1672831391552",
"Statement": [
{
"Sid": "Stmt1672830972647",
"Effect": "Allow",
"Principal": "",
"Action": "s3:
",
"Resource": "arn:aws:s3:::XXXXX/"
},
{
"Sid": "Stmt1672831390540",
"Effect": "Allow",
"Principal": "
",
"Action": "s3:*",
"Resource": "arn:aws:s3:::XXXXX"
}
]
}

Collapse
 
kasukur profile image
Sri

@anusachdeva4321 ,

you don't need S3 bucket to be public, actually you should never make your bucket public without a proper bucket policy.

We create a user with programmatic access and we use the use at Step 9 of Configure Jenkins. which will be used when you run the pipeline.

PLUS

Step 6. Create a Pipeline
Navigate to Dashboard > Manage Jenkins and copy the ID of IAM User created in Step 1, which we need to replace xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the ID.

There is an option to perform this with EC2 Instance role but that option has not been covered in this blog post.

hope I have answered your questions,
Sri

Collapse
 
anusachdeva4321 profile image
Anu Sachdeva

I am trying to understand why I got the 403 , I will try this again by following this tutorial and share the error if I get it again.

Collapse
 
mlops_engineer profile image
John🪙🇬🇧

I followed the steps through and was able to get it done. Thanks for the time you spent on making this post.

Collapse
 
kasukur profile image
Sri

Happy that my post helped you @mlops_engineer.

Collapse
 
anusachdeva4321 profile image
Anu Sachdeva

When you mention ID instead of xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, is it the fully qualified ARN "arn:aws:iam::AWS_ACT:user/JenkinsUser" or is it only "JenkinsUser" ?

Collapse
 
kasukur profile image
Sri • Edited

The id of the user created in step 1, which will be in the Jenkins console.
I haven’t provided the screenshot for security reasons.

Collapse
 
mfschacht profile image
Matthew Schacht

Hello. Thank you for this article, as it's been very helpful in setting up my own Jenkins controller in an EC2 instance.
I am having an issue getting builds to run, using the dynamically allocated cloud nodes (i.e., spinning up other EC2 instances as nodes for Jenkins to run builds). I have followed all the instructions on this guide up to step 6 (I am going to setup a different pipeline for my repo), other than attached a more permissive S3 permission to the IAM role so it can write to buckets as well, and I chose the Amazon Linux 2023 AMI because the one listed in these instructions is deprecated.
I have disabled the built-in node because I want to dynamically allocate nodes (and the instance does not have enough space in the /tmp in order to run said node anyway, and it seems difficult to change that). I setup the Cloud nodes as instructed here, but when I start a build, the build just hangs and says its waiting for an available node. Is there some step I'm missing, or perhaps something that was default when these instructions were written that must now be done manually?