Optimizing GitLab CI for Readability and Maintainability: From 1K to 600 Lines!

This short article aims to share some tips I learnt from optimizing our GitLab CI file (.gitlab-ci.yaml) for building multiple Docker images.

By implementing these techniques, you’ll not only make your GitLab CI files more readable but also set the stage for a more agile and adaptable CI/CD environment.
Let’s delve into the practical insights that can elevate your DevOps experience and contribute to a more efficient and maintainable codebase.

1- Use default runner tag and override it when needed:

In case you are using a standard or a generic runner for the majority of your jobs and a a different runner for a few jobs, consider defining a global runner tag instead of defining one for each job.
You can then override the global tag when necessary.

❌ Don’t do:

compile:
  script:
    - ...
  tags:
    - myruuner
test:
  script:
    - ...
  tags:
    - testrunner  
deploy:
  script:
    - ...
  tags:
    - myrunner

✅ Do:

default:
  tags:
    - myrunner

compile:
  script:
    - ...

test:
  script:
    - ...
  tags:
    - testrunner  
deploy:
  script:
    - ...

2- Use parallel:matrix

GitLab has a powerful CI feature to run a matrix of jobs in parallel.
In our case we had multiple jobs to test every customized Docker JDK image and ensure that it does not break a standard maven build.
Taking advantage of the keyword parallel:matrix, can save you multiple lines of code and make your CI files more readable and easier to maintain.

Here is an example of refactoring our test jobs:

❌ Instead of configuring multiple job with the same script part, for example:

test-mvn-jdk-11:
  image: openjdk-11
  stage: test-image
  script:
    - mvn -V compile ...

test-mvn-jdk-17:
  image: openjdk-17
  stage: test-image
  script:
    - mvn -V compile ...

test-mvn-jdk-21:
  image: openjdk-21
  stage: test-image
  script:
    - mvn -V compile ...
....

✅ You can easily replace them with:

test-mvn:
  image: $JAVA_IMAGE
  stage: test-image
  script:
    - mvn -V compile ...
  parallel:
    matrix:
      - JAVA_IMAGE: [openjdk-11,openjdk-17,openjdk-21]

This will also make it easier to incorporate additional JDK versions in the future, requiring minimal adjustments and eliminating the need for extensive line changes or introducing new jobs.

3- Use rules to define variables:

GitLab rules provide a mechanism for specifying conditions to determine when CI/CD jobs run.
Additionally, the use of if statements within these rules allows for the dynamic definition of variables based on conditions such as branches or releases, offering flexibility in variable assignment.

For our repository's CI/CD workflow, we start by building docker images and test them if there's a Merge Request.
Only when everything checks out, we create a final version TAG for release.
Using rules reduced our build jobs by a half.

❌ Instead of defining two jobs, for example:

build-mr-jdk-11:
  stage: build
  script:
    - docker build -t jdk-11:mr-validation
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

build-jdk-11:
  image: openjdk-11
  stage: build
  script:
    - docker build -t jdk-11:$CI_COMMIT_TAG
  rules:
    - if: $CI_COMMIT_TAG

✅ Use the same job and override the variables depending on the rules conditions:

build-jdk-11:
  stage: build
  script:
    - docker build -t jdk-11:$TAG
rules:
    - if: $CI_COMMIT_TAG
      variables:
        TAG: "$CI_COMMIT_TAG"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      variables:
        TAG: "mr-validation"

Conclusion: Lean back and let the magic happen 🧙‍♀

In summary, applying these tips reduced our GitLab CI file from 1176 to 667 lines, significantly improving efficiency, manageability, and overall workflow.

⭐⭐⭐ Enjoy your learning!!! ⭐⭐⭐