Building an Enterprise CI/CD Pipeline with Jenkins, Docker, Trivy, and GKE

Jenkins, the most popular open-source tool for build automation, plays a vital role in efficiently building, testing, and rolling out new applications across various environments.

One of those environments is Kubernetes, which is widely available as a managed service from all the leading cloud providers. In this article, we will explore how to build an automated, enterprise-ready CI/CD pipeline using Jenkins, Docker, and GitHub, and deploy our application to a Kubernetes Cluster on Google Kubernetes Engine (GKE).

Prerequisites

• We need a running Jenkins server with SSH access to it. For this project, I have used Compute Engine to spin up an Ubuntu instance for me.
• We need an active GCP Project and required access to it.
• We have a Kubernetes cluster running on Google Kubernetes Engine (GKE). If you are new to GCP, You can use free-tier credits offered by the GCP for this demo.
• We have a GitHub account for our code repository.

Pipeline Architecture Overview

Here’s a high-level look at the CI/CD pipeline that we'll be building:

• Source Code Management: GitHub will serve as the version control system, where changes in the codebase trigger the pipeline execution.
• Continuous Integration: Jenkins will be used to automate the entire build and deployment workflow. We'll also incorporate Trivy for security scans.
• Continuous Deployment: The application will be built as a Docker image, pushed to Google Artifact Registry, and then deployed to GKE.
• Monitoring and Notifications: Prometheus and Grafana will be set up to continuously monitor the health and performance of the deployed application. Email notifications will be configured to alert the team about build statuses and vulnerability scans.

Let's Dive in !

Setting Up Jenkins Server

Fire up an Ubuntu instance with a recommended disk space of 15GB and above.

Once the instance is up and running, SSH into it.

Run these commands on the machine to install Jenkins, Docker, Trivy and Kubectl

wget https://raw.githubusercontent.com/akshat2503/Jenkins-CI-CD/main/jenkins-server-setup.sh
chmod +x jenkins-server-setup.sh
./jenkins-server-setup.sh

I am doing this tutorial in an Ubuntu Instance. For other Linux distributions, installation steps may differ. Please follow the official documentation Jenkins Installation

After installation completes, The output will be something like this:

Access the Jenkins server on the given link & authenticate using the first time password.

Install all suggested plugins

and finally create the admin user.

Prepare Jenkins

• Install Required Plugins from Dashboard > Manage Jenkins > Plugins

• Create credentials in Dashboard > Manage Jenkins > Credentials

1) Service Account from GCP

Create a service account, assign necessary permissions (like Artifact Registry Administrator, Kubernetes Engine Admin, Service Account User, etc.)

Create a key for it, and it will be downloaded on your local machine.

Save the credential in Jenkins. Choose kind as "Secret File" & upload the downloaded key.

2) Google App Password

Go to Manage Google Account > Security > 2-Step verification > App passwords

Create a new app password and copy it somewhere

Set username as the email address of the account from which the app password was generated & the password as the copied app password.

Explaining Pipeline Steps

1) Authenticate with Google Cloud

This step will use gcloud SDK to activate the service account & authorize Docker using the key you provided in Jenkins credentials.

withCredentials([file(credentialsId: 'service-acc-cred', variable: 'GC_KEY')]) {

    sh '''

        gcloud auth activate-service-account --key-file="$GC_KEY"

        gcloud config set project $PROJECT_ID

        gcloud auth configure-docker gcr.io -q

    '''

}

2) Clean Workspace

Cleans the workspace by deleting all the files from previous builds.

deleteDir()

3) Checkout

Clones the GitHub repository to the Jenkins server for further processing.

sh 'git clone https://github.com/akshat2503/akshatgautam'

4) File System Scan

Scans the downloaded GitHub repository for any possible vulnerability using Trivy and generates a report for it.

sh "trivy fs --format table -o trivy-fs-report.html ."

5) Build Docker Image

Uses the Dockerfile present in the repository to build a Docker image.

You can also tag the image with some other parameter such as BUILD_NUMBER, I used latest just for simplicity.

sh 'cd akshatgautam && docker build -t gcr.io/$PROJECT_ID/portfolio-website:latest .'

6) Docker Image Scan

Scans the generated Docker image for any possible vulnerability using Trivy and generates a report for it.

sh "trivy image --format table -o trivy-report.html gcr.io/$PROJECT_ID/portfolio-website:latest"

7) Push Docker Image

The Docker image is then pushed to Google Artifact Repository.

sh 'docker push gcr.io/$PROJECT_ID/portfolio-website:latest'

8) Deploy to GKE

This step fetches credentials for connection to your GKE cluster and then applies a deployment.yaml file.

The YAML file creates a deployment and a service of LoadBalancer type that will serve the application.

withCredentials([file(credentialsId: 'service-acc-cred', variable: 'GOOGLE_APPLICATION_CREDENTIALS')]) {

    sh """

        export PATH=$PATH:$HOME/bin

        gcloud config set project "$PROJECT_ID"

        gcloud container clusters get-credentials "$CLUSTER_NAME" --zone "$CLUSTER_ZONE"

        kubectl apply -f akshatgautam/k8s/deployment.yaml

    """

}

9) Verify Deployment

Verifies the deployment using kubectl and logs the output to console.

sh 'kubectl get services'

9) Email the final status

The post script sends an email with the status of the pipeline execution and also attaches both of the Trivy reports with the email.

Pipeline Creation

Let's get started with setting up your Jenkins pipeline:

• Go to Dashboard > New Item

Name: jenkins-gcp-integration-pipeline
Project Type: Pipeline

• Scroll down and add the pipeline script, Get the script from here

Our pipeline is set up, but not yet ready to build. Mail configuration to receive mail updates is still pending !

• Go to Dashboard > Manage Jenkins > System, scroll down to find Extended E-mail Notification

SMTP Server: smtp.gmail.com
SMTP Port: 465
Credentials: Choose the one we previously created
Check "Use SSL"

• Now just below this, You'll find E-mail Notification

SMTP Server: smtp.gmail.com
Check "Use SMTP Authentication"
Username: Your email
Password: App Password that we copied earlier
Check "Use SSL"
SMTP Port: 465

Click "Save"

Voila! Your Jenkins pipeline is now ready

Pipeline Testing

• Go to Dashboard > jenkins-gcp-integration-pipeline and click Build Now

A build will trigger within a second. All the pipeline steps will be followed sequentially.

Here are the results:

Verifying Deployment

• Go to GKE > Workloads

• Go to GKE > Gateways, Services & Ingress. Your application should be listed there.

Conclusion

At this point, we have successfully created a simple CI/CD pipeline between Jenkins, GitHub, Artifact Registry and one Kubernetes cluster running on Google Kubernetes Engine.

You can further enhance this pipeline by adding some features like:

• GitSCM Polling/Webhook to trigger build on code commit
• Including integration, performance tests on the code
• Multi-cluster deployment
• Error handling and retry mechanisms
• and what not ...

Monitoring through Prometheus & Grafana will be documented in another article to keep this one concise.

Drop your questions in comments and I will be very happy to help !

Happy Learning !