Two ways to keep gitlab CI files maintainable

Once we had a gitlab CI file. It was short and sweet. One year later, it grew 350 lines long.

There were these problems:

1. Too much content - too much scrolling, hard to visualize and work on.
2. Hard to disable some jobs temporarily - mostly for debugging infra or test environment or emergency deployment (let that never happen!).

Let's try to solve it with some features from gitlab.

1. Leverage templating

Gitlab CI supports using templates within the .gitlab-ci.yml file.

Consider a sample .gitlab-ci.yml file as follows

stages:
   - setup
   - soft-qa //lint and unit tests
   - build
   - hard-qa //e2e's
   - deploy-storybook
   - pack
   - notify-devs-staging-can-be-deployed
   - deploy-staging
   - notify-devs-prod-can-be-deployed
   - deploy-production
   - suggest-release-notes

variables: 
    var1: '1'
    // .. and so on

conditions:
    only_master: 
       // configs
    branches:
       // configs
    // and more...

## cache related configs

## setup related configs

## jobs for each for the stages start
.
.
.
. // 200 lines later
.
## jobs end

I will still keep the stages in the .gitlab-ci.yml untouched so that I can get a preview of all the stages right at the start of the file.

Let's now split this into smaller templates.

For templates, I will create a folder in the repo root called ci-templates. Now let's extract out one job from the main file and place it in a template in this folder. Note, all these files has to YAML, to be included into a .gitlab-ci.yml file.

/ci-templates/.soft-qa.yml

soft-qa:
  image: node:14.5
  stage: soft-qa
  <<: *npm_cache_pull
  allow_failure: false
  script:
    - yarn lint
    - yarn test:unit:ci
  artifacts:
    paths:
      - coverage/lcov.info

Time to use the template. Go to .gitlab-ci.yml and include this file like below. I will place it at the position of the replaced job.

include: '/ci-templates/.soft-qa.yml'

We are using this syntax - with relative path - as we are using the file from the same repo. You can keep the file in another repo on the same gitlab instance or even in a public remote repository and use it! Just look up the syntax and update accordingly, like below:

For another repo:

include:
  - project: 'my-space/my-another-project'
    file: '/templates/.build-template.yml'

For remote:

include:
  - remote: 'https://somewhere-else.com/example-project/-/raw/master/.build-template.yml'

In a similar way, extract out other jobs as well and remove them from the .gitlab ci file. I have abstracted the notify stages to a file called .notifications.yml and deployment related jobs to .deploy.yml, thus separating the concerns from one single file. Now, include becomes a list like below:

include: 
   - '/ci-templates/.soft-qa.yml'
   - '/ci-templates/.build.yml'
   - '/ci-templates/.hard-qa.yml'
   - '/ci-templates/.deploy-storybook.yml'

When pipeline starts, all included files are evaluated and deep merged into the .gitlab-ci file.

Catch, Catch, Catch

Things are getting interesting. I have certain conditions like only_tag, only_branches in these jobs. How would I provide them to these files without duplicating it in every files?

Enter extends.

I will consolidate my conditions in a file, say, .conditions.yml.

// /ci-templates/.conditions.yml

.only_tag:
  only:
    - /^v\d+\.\d+\.\d+$/
  // and more...

To use it in a template, include it first in the .gitlab-ci file

include: 
   - '/ci-templates/.conditions.yml'
   - '/ci-templates/.soft-qa.yml'
   - '/ci-templates/.build.yml'
   - '/ci-templates/.hard-qa.yml'
   - '/ci-templates/.deploy-storybook.yml'

Then, in the template file, add extend. Extend is a way of extending the congfigurations to another file. YAML anchors can be used but only from the same file. So this is a way to use jobs from another template.

deploy_staging:
  extends: .only_tag
  <<: *deploy

Now, deploy_staging will be created only for tags.

Finally, the .gitlab-ci file will look something like:

stages:
   - setup
   - soft-qa //lint and unit tests
   - build
   - hard-qa //e2e's
   - deploy-storybook
   - pack
   - notify-devs-staging-can-be-deployed
   - deploy-staging
   - notify-devs-prod-can-be-deployed
   - deploy-production
   - suggest-release-notes

variables: 
    var1: '1'
    // .. and so on

## cache related configs

## setup related configs

include: 
   - '/ci-templates/.conditions.yml'
   - '/ci-templates/.soft-qa.yml'
   - '/ci-templates/.build.yml'
   - '/ci-templates/.hard-qa.yml'
   - '/ci-templates/.deploy-storybook.yml'
   - '/ci-templates/.pack.yml'
   - '/ci-templates/.notifications.yml'
   - '/ci-templates/.deploy.yml'
   - '/ci-templates/.suggest-release-notes.yml'

Now let's go back to the initial problems and check whether they are fixed:

1. Too much content to scroll - now, we can view whole content in the IDE window - Fixed
2. Hard to disable jobs - now, we just have to comment the template in the include list - Fixed

How will I know, if I commented out a required job? Like, by mistake, I commented the pack job. For deployment packed resources are necessary. Here, we will know that the deployment failed much later when the deployment runs. To solve this, there is a way to mark dependencies on previous jobs. That is explained in next step.

2. Use needs or dependencies as applicable

One way to track that you have the dependent artifacts are available before starting a job is by using dependencies or
needs keyword.

By default, all artifacts from previous stages are passed to each job. However, you can use the dependencies keyword to define a limited list of jobs to fetch artifacts from.

build_frontend:
   stage: build
   script: yarn build

deploy:
   stage: deployment
   dependencies: 
      - build_frontend

In this case, if the build_frontend is not available, while merging the templates, the pipeline will report error that the build_frontend is not available. Easy to understand.

Another utility is the needs. It is mainly used when you are running jobs out of order and still ensure that the dependency job is completed before starting a job. The difference of needs from dependencies is that, in needs, no artifacts from previous steps are downloaded by default. You need to use artifacts: true config on the job like below to download them:

deploy:
  needs:
   - job: build_1
     artifacts: true
   - job: build_2
     artifacts: false

This helped me reduce the complexity of the CI file. Hope it helps you too.

Peace ✌️