Automate Jenkins Job builds with Python

We often come across a situation when we have to manually run a Jenkins job many times each possibly having different parameters. One way to do it is using the Jenkins UI and pressing the build with parameters button each time and passing on the parameter values each time. This can be quite tiresome and repetitive. Fortunately, I am too Lazy to do it like that xD. So, I found another way, we can trigger the Jenkins jobs using the Jenkins REST API. In this post, we will demonstrate how to design our Jenkins jobs effectively for this purpose and use python to make use of the Jenkins API to automate triggering a huge number of jobs and then take a chill pill :)

Step 1: Enable Triggering Jenkins jobs from remote scripts

Navigate to your Jenkins Job configuration page and scroll down to the Build Triggers option. Enable the Trigger builds remotely (e.g., from scripts) flag and put some secret string in the Authentication Token field.

Step 2: Parameterize Job -> GIT SCM (BRANCH, REPO ETC.)

We need to make our Job bash script read from parameters we passed in to the Job so that we can easily control these parameters from our python script. Furthermore, if there's a git repo/branch our Job needs to checkout we can easily control that from python as well.
Enable the This project is parameterized flag from the Jenkins job configuration page and add the required parameters like below,

Jenkins supports various different types of parameters including Strings, Boolean, Choices, File etc. Make sure you are choosing them appropriately for your choice. In my case for example, I had to deploy a bunch of Ansible playbooks to a Kubernetes cluster so, I needed the following parameters:

PLAYBOOK - name of the yaml ansible playbook file
GIT_REPO_ID - git repo to checkout
GIT_BRANCH_NAME - branch name of the repo (particularly useful when you are testing/reviewing something on a feature branch)
HELM_REPOSITORY - helm repository name that my ansible playbooks needs

Now, access these parameters wherever you need inside the Jenkins config using the ${PARAM_NAME} syntax.

For example my job bash script looks like the one just below where, I am utilizing the PLAYBOOK and HELM_REPOSITORY parameters.

#!/bin/bash
set -xe

pip3 install --quiet -r requirements.txt

cd ansible

ANSIBLE_FORCE_COLOR=true ansible-playbook -e "helm_repository=${HELM_REPOSITORY}" \
playbooks/${PLAYBOOK} -vvvv

Here's another example where I am utilizing the GIT SCM parameters,

Step 3: Generate Jenkins Crumb

Jenkins API will need us to pass in a Jenkins-Crumb header to our requests. Therefore, we need to generate this crumb value from the Jenkins CrumbIssuer before making the actual build trigger requests. Here's how we can generate the Jenkins Crumb using python:

def jenkins_crumb():
    session = requests.Session()
    session.auth = ("<JENKINS_USERNAME>", "<JENKINS_USER_PASSWORD>")
    crumb = session.get("https://<YOUR_JENKINS_BASE_URL>/crumbIssuer/api/json")
    return crumb.request.headers["Authorization"], crumb.json()["crumb"]

Step 4: Trigger Jenkins Job from Python

Now, we can finally trigger our job from python scripts. Here's an example,

TOKEN = "<YOUR_AUTH_TOKEN>" # the token value you set in STEP-1

def deploy_service(
    service_name,
    playbook,
    branch="master",
    repo="ashiqursuperfly/jenkins-job-config",
    chart_repo="ashiqursuperfly-portfolio-dev"
):
    JOB_URL = f"https://<YOUR_JENKINS_BASE_URL>/job/<YOUR_JENKINS_JOBNAME>/buildWithParameters?token={TOKEN}"
    url = f"{JOB_URL}&PLAYBOOK={playbook}&GIT_BRANCH_NAME={branch}&GIT_REPO_ID={repo}&HELM_REPOSITORY={chart_repo}"

    cookie, crumb = jenkins_crumb()

    headers = {"Jenkins-Crumb": crumb}
    session = requests.Session()
    session.auth = ("<JENKINS_USERNAME>", "<JENKINS_USER_PASSWORD>")

    res = session.get(url=url, headers=headers)

    if res.ok:
        print(f"Started deploy job for service {service_name} with playbook {playbook}")
    else:
        print(
            f"Error deploy job for service {service_name} with playbook {playbook}\n{res.json()}"
        )

if __name__ == "__main__":
    list_of_service = list_services()

    for service in list_of_service:
        res, playbook = get_playbook_name(service)
        deploy_service(service, playbook)
        sleep(100)

Now, we can invoke the following method as many times as we want. In my case, I just looped through my list of services to deploy and invoked this method for each service and just watched Jenkins complete all the tedious work for me as I sipped my coffee, basked in the brightness of my monitor's light, and wondered why life is so simple when you can't stop thinking automation 🤓

If you found this post helpful, CLICK BELOW 👇