Table of Contents
- Introduction
- Create an IAM User
- Create a Key pair
- Create a Security Group
- Create an EC2 instance
- Install and Configure Jenkins
- Create a Pipeline
- Clean Up
- Summary
- Referrals
Introduction
In this blog we are going to set up Jenkins on an EC2 instance and then set up a pipeline to copy a file from S3.
We are going to start something very basic and enhance it as we move along.
What is Jenkins?
Jenkins offers a simple way to set up a continuous integration or continuous delivery (CI/CD) environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks. While Jenkins doesn’t eliminate the need to create scripts for individual steps, it does give you a faster and more robust way to integrate your entire chain of build, test, and deployment tools than you can easily build yourself.
What is Jenkins Pipeline?
Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.
Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines as code
via the Pipeline domain-specific language (DSL) syntax.
Demo
Let's get started with the demo.
Step 1. Create an IAM User
- Navigate to IAM > Users > Click on Add users
- Enter User Name as JenkinsUser
- Select Access key - Programmatic access and Click Next: Permissions
- Click Attach existing policies directly and then Create policy
- In the Create policy window, Click on JSON and add the following JenkinsEC2Policy and save the policy as JenkinsEC2Policy
- Select AmazonS3ReadOnlyAccess and JenkinsEC2Policy under Attach existing policies directly
- Click Next:Tags, Next:Review and then Create user
-
Download the credentials .csv file as we will need this during Jenkins configuration at Step 6.
JenkinsEC2Policy
{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1312295543082", "Action": [ "ec2:DescribeSpotInstanceRequests", "ec2:CancelSpotInstanceRequests", "ec2:GetConsoleOutput", "ec2:RequestSpotInstances", "ec2:RunInstances", "ec2:StartInstances", "ec2:StopInstances", "ec2:TerminateInstances", "ec2:CreateTags", "ec2:DeleteTags", "ec2:DescribeInstances", "ec2:DescribeInstanceTypes", "ec2:DescribeKeyPairs", "ec2:DescribeRegions", "ec2:DescribeImages", "ec2:DescribeAvailabilityZones", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "iam:ListInstanceProfilesForRole", "iam:PassRole", "ec2:GetPasswordData" ], "Effect": "Allow", "Resource": "*" } ] }
IAM_JenkinsUser1
IAM_JenkinsUser2
IAM_JenkinsUser3
Step 2. Create a key pair
- Navigate to EC2 > Key Pairs (under Network & Security).
- Click
Create key pair
. - Navigate to the folder where the key pair is downloaded and run.
chmod 400 <key_pair_name>.pem
Jenkins_Keypair
Step 3. Create a Security Group
- We are going to create a Security Group for SSH and Jenkins web access.
- Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
- Name:
JenkinsSG
. - Add an Inbound rule to allow
SSH (TCP 22)
traffic fromMy IP
. - Add another Inbound rule to allow
Custom (TCP 8080)
traffic fromMy IP
.
- Name:
Jenkins_SG
Step 4. Create an EC2 instance
- Navigate to EC2 > EC2 Dashboard > Click on Launch instance.
- Launch an instance with the following values as shown in the screenshots.
- When the instance state is Running, select the instance and click on Connect and then copy the connections details.
-
Connect to the instance.
ssh -i "Jenkins-Keypair.pem" ec2-user@ec2-54-211-70-130.compute-1.amazonaws.com
Jenkins_EC1
Jenkins_EC2
Step 5. Install and Configure Jenkins
Install Jenkins
-
Ensure that the software packages are up to date on the instance by executing the following command:
[ec2-user ~]$ sudo yum update –y
-
Add the Jenkins repo using the following command:
[ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
-
Import a key file from Jenkins-CI to enable installation from the package:
[ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key [ec2-user ~]$ sudo yum upgrade
-
Install Java:
[ec2-user ~]$ sudo amazon-linux-extras install java-openjdk11 -y
-
Install Jenkins:
[ec2-user ~]$ sudo yum install jenkins -y
-
Enable Jenkins service to auto start at boot:
[ec2-user ~]$ sudo systemctl enable jenkins
-
Start Jenkins as a service:
[ec2-user ~]$ sudo systemctl start jenkins
-
You can check the status of the Jenkins service using the command:
[ec2-user ~]$ sudo systemctl status jenkins
-
Just in case we ever need to restart Jenkins during this setup/configuration.
[ec2-user ~]$ sudo systemctl restart jenkins
Configure Jenkins
- Copy the Public IPv4 DNS of EC2 instance and paste the URL as following in the browser For Example: http://ec2-54-211-70-130.compute-1.amazonaws.com:8080
-
Enter the initialAdminPassword from
/var/lib/jenkins/secrets/initialAdminPassword
and Click Continue
[ec2-user@ip-172-31-89-157 ~]$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword 671x0x5x3xxx46xxxxxx099x1xf0149x
Select Install suggested plugins.
Once the installation is complete, Create First Admin User will open. Enter your information, and then select Save and Continue.
Click Dashboard, select Manage Jenkins, and then select Manage Plugins.
Click Available plugins, Search and Select Amazon EC2 and then **Install without restart*
Once the installation is complete, Navigate back to Dashboard, select Manage Jenkins, select Manage nodes and clouds and then click on Configure Clouds.
Select Add a new cloud, and select Amazon EC2. A new pop up window opens with more fields.
-
Click Add under Amazon EC2 Credentials and Select Jenkins
- From the Jenkins Credentials Provider: Jenkins, select AWS Credentials as the Kind.
- Enter Access Key ID, Secret Access Key from the key pair and Click Add.
Scroll down to Region and select your Region.
-
Click Add under EC2 Key Pair's Private Key and Select Jenkins.
- From the Jenkins Credentials Provider: Jenkins, select
SSH Username with private key
as the Kind and set the Username toec2-user
. - Select
Enter Directly
under Private Key, then select Add. - Open the private key pair you created in the creating a key pair step and paste in the contents from
-----BEGIN RSA PRIVATE KEY-----
to-----END RSA PRIVATE KEY-----
. Select Add when completed.
- From the Jenkins Credentials Provider: Jenkins, select
Scroll down to Test Connection and ensure it states
Success
and then Click Save.
Jenkins_Configure1
Jenkins_Configure2
Jenkins_Configure3
Jenkins_Configure4
Jenkins_Configure5
Jenkins_Configure6
Jenkins_Configure7
Jenkins_Configure8
Jenkins_Configure9
Jenkins_Configure10
Jenkins_Configure10.1
Jenkins_Configure10.2
Jenkins_Configure11
Jenkins_Configure11.1
Jenkins_Configure11.2
Jenkins_Configure12
Step 6. Create a Pipeline
- Navigate to Dashboard > Manage Jenkins and copy the ID of IAM User created in Step 1, which we need to replace
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
with the ID. - Navigate to Dashboard > Manage Jenkins > Plugin Manager > Click on Available plugins
- Search for AWS Steps and Install without restart.
- Navigate to Dashboard > New Job > Enter
download-a-file-from-s3
> Select Pipeline and Click OK. - Scroll down to Pipeline, add the following
Pipeline script
and Click Save - Create an S3 bucket and copy some files to the bucket.
- Replace the
s3bucket
with your bucket name andfilename
with one of the files in your S3 bucket. - Navigate to Dashboard > download-a-file-from-s3 and Click on Build Now.
-
Navigate to the latest Build History link and check out the Console Output.
pipeline { agent any stages { stage('S3download') { steps { withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\ { s3Download(file: "filename", bucket: 's3bucket', path: '') } } } } }
Jenkins_Pipeline1
Jenkins_Pipeline2
Jenkins_Pipeline3
Jenkins_Pipeline4
The following is the console log, which shows the pipeline has been successful. ✅
Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh
Finished: Downloading from s3bucket/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
The file that was copied from S3 to the EC2 instance. ✅
[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/
total 0
drwxr-xr-x 2 jenkins jenkins 55 Dec 31 03:45 receiveMessages.sh
Let's run the build again, this time the build has failed with the following error ❓
The error message suggests to use set force=true
.
Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/
Download failed due to existing target file; set force=true to overwrite target file
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Target exists: file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/
at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:146)
at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:113)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE
So we need to add force:true
as per the Pipeline: AWS Steps documentation
pipeline
{
agent any
stages
{
stage('S3download')
{
steps {
withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
{
s3Download(file: "filename", bucket: 's3bucket', path: '', force:true)
}
}
}
}
}
The build succeeded after adding force:true
and now we can run the build multiple times.
Now let's parameterise the destination foldername.
- Navigate to Dashboard > download-a-file-from-s3 and click on Configure
- Select This project is parameterised and then Add String parameter
- Enter Name as
foldername
, Default Valu asfoldername
and Save - Click Build with Parameters
- Navigate to the latest Build History link and check out the Console Output.
pipeline
{
agent any
stages
{
stage('S3download')
{
steps {
withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
{
echo "${foldername}"
s3Download(file: "${foldername}", bucket: 's3bucket', path: '', force:true)
}
}
}
}
}
Jenkins_Pipeline_Parameter
Jenkins_Pipeline_Build-with-Parameters
The following is the console log, which shows the pipeline has been successful. ✅
Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1
[Pipeline] {
[Pipeline] echo
s3files
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/s3files
Finished: Downloading from sqssri/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
The files were copied to s3files directory on the EC2 instance. ✅
[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/s3files
total 8
-rw-r--r-- 1 jenkins jenkins 952 Dec 31 03:40 receiveMessages.sh
-rw-r--r-- 1 jenkins jenkins 646 Dec 31 03:40 sendMessages.sh
Clean Up
- Delete the
EC2
instance. - Delete the
JenkinsUser
. - Delete the
key pair
. - Delete the Security Group
JenkinsSG
.
Summary
- We learned how to install and configure Jenkins.
- We also learned about setting up the pipeline.
Top comments (8)
Thank you for the descriptive article. I have a few questions please 1) Why was the S3 bucket not made public and no bucket policy attached to let Ec2 access this s3.
2) why was no role attached to the ec2 to access the S3 bucket?
I had to follow some more additional steps to complete this setup. I had to do below coz I was getting 403 access denied error. I am not sure which one step exactly worked because I feel it is cumulation of all these steps.
1) Made my s3 bucket public 2) Attached below policy to my S3 bucket . 3) Attached an IAM role to my Jenkins ec2 so that it can access s3(not sure if this was actually needed or not but I tired everything to resolve the 403 error I was getting) . After following these steps my 403 error message went away and the job build was success
{
"Version": "2012-10-17",
"Id": "Policy1672831391552",
"Statement": [
{
"Sid": "Stmt1672830972647",
"Effect": "Allow",
"Principal": "",
"Action": "s3:",
"Resource": "arn:aws:s3:::XXXXX/"
},
{
"Sid": "Stmt1672831390540",
"Effect": "Allow",
"Principal": "",
"Action": "s3:*",
"Resource": "arn:aws:s3:::XXXXX"
}
]
}
@anusachdeva4321 ,
you don't need S3 bucket to be public, actually you should never make your bucket public without a proper bucket policy.
We create a user with programmatic access and we use the use at Step 9 of Configure Jenkins. which will be used when you run the pipeline.
PLUS
Step 6. Create a Pipeline
Navigate to Dashboard > Manage Jenkins and copy the ID of IAM User created in Step 1, which we need to replace xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the ID.
There is an option to perform this with EC2 Instance role but that option has not been covered in this blog post.
hope I have answered your questions,
Sri
I am trying to understand why I got the 403 , I will try this again by following this tutorial and share the error if I get it again.
I followed the steps through and was able to get it done. Thanks for the time you spent on making this post.
Happy that my post helped you @mlops_engineer.
When you mention ID instead of xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, is it the fully qualified ARN "arn:aws:iam::AWS_ACT:user/JenkinsUser" or is it only "JenkinsUser" ?
The id of the user created in step 1, which will be in the Jenkins console.
I haven’t provided the screenshot for security reasons.
Hello. Thank you for this article, as it's been very helpful in setting up my own Jenkins controller in an EC2 instance.
I am having an issue getting builds to run, using the dynamically allocated cloud nodes (i.e., spinning up other EC2 instances as nodes for Jenkins to run builds). I have followed all the instructions on this guide up to step 6 (I am going to setup a different pipeline for my repo), other than attached a more permissive S3 permission to the IAM role so it can write to buckets as well, and I chose the Amazon Linux 2023 AMI because the one listed in these instructions is deprecated.
I have disabled the built-in node because I want to dynamically allocate nodes (and the instance does not have enough space in the /tmp in order to run said node anyway, and it seems difficult to change that). I setup the Cloud nodes as instructed here, but when I start a build, the build just hangs and says its waiting for an available node. Is there some step I'm missing, or perhaps something that was default when these instructions were written that must now be done manually?