DEV Community

Cover image for 🦊 GitLab CI YAML Modifications: Tackling the Feedback Loop Problem
Benoit COUETIL 💫 for Zenika

Posted on • Updated on

🦊 GitLab CI YAML Modifications: Tackling the Feedback Loop Problem

Initial thoughts

As developers and CICD engineers, we are all too familiar with the time-consuming process of iterating on GitLab CI YAML modifications. It can be a frustrating cycle that involves multiple steps and setbacks:

  • Make a change to the YAML file.
  • Commit and push the changes.
  • Verify that the CI YAML is valid and the graph is as expected.
  • If there are any issues or errors, start the process all over again with a sense of despair.
  • Wait for the entire pipeline to finish running on the current branch.
  • If the pipeline fails or doesn't behave as desired, start over again with a heavy heart.
  • Refactor commits to maintain a clean git graph and push with force.
  • Merge multiple times to observe the behavior on long-lived branches and tags.
  • If there are any issues with the merged code, start the process all over again with a sigh.

What we initially thought would only take a few minutes ends up consuming multiple days as we continuously go back and forth, all while trying to juggle other development tasks.

In this article, we aim to alleviate this pain by providing several tips that will make your life easier. Our goal is to transform the experience of tweaking your favorite pipelines from a tedious chore to an enjoyable task. So let's dive in and discover how you can streamline your GitLab CI YAML modifications!

1. Use a local docker image in early stages of job creation

When starting a new job with a new Docker image, it's natural to feel uncertain. Especially when using initially unknown public images, following the best practice Start CI with versioned public CI docker images. To accelerate the feedback loop and gain more confidence, we can try our commands directly in a local container.

Let's consider a NodeJS job as an example. The DockerHub image alpine/k8s seems to be the most suitable base image.

Before diving into the GitLab CI YAML, let's start an interactive container:

docker run -it --privileged=true node:20.5-alpine3.17 /bin/sh
Status: Downloaded newer image for node:20.5-alpine3.17
Enter fullscreen mode Exit fullscreen mode

You may wonder what shell GitLab is using in containers. In fact, it is a discovery process starting with bash, so best guess is to try /bin/bash and fallback to /bin/sh.

Now that you're inside the image, you can easily prepare and test your commands.

Although the base is alpine, let's assume we do not have that knowledge and want to use envsubst. If it's not available, we can try to install it with apt-get update && apt-get install envsubst:

/# envsubst --help
/bin/sh: envsubst: not found
/# apt-get update && apt-get install envsubst
/bin/sh: apt-get: not found
Enter fullscreen mode Exit fullscreen mode

Since envsubst and apt-get are not available, it must be the alpine Linux package manager, apk:

/# apk --update add --no-cache envsubst
ERROR: unable to select packages:
  envsubst (no such package):
    required by: world[envsubst]
Enter fullscreen mode Exit fullscreen mode

Ah! It seems that envsubst cannot be installed this way. A quick internet search reveals that the package is actually named gettext:

/apps# apk --update add --no-cache gettext
(1/9) Installing libgomp (12.2.1_git20220924-r4)
(8/9) Installing libxml2 (2.10.4-r0)
(9/9) Installing gettext (0.21.1-r1)
Executing busybox-1.35.0-r29.trigger
OK: 18 MiB in 26 packages
/# envsubst --help
Usage: envsubst [OPTION] [SHELL-FORMAT]

Substitutes the values of environment variables.
Enter fullscreen mode Exit fullscreen mode

By going through this discovery process within a few minutes, we saved ourselves from waiting for the pipeline to fail, which could have taken several hours for a job, especially positioned in last stages.

2. Use the pipeline editor for minor YAML changes

The GitLab CI/CD Pipeline Editor is a powerful web-based tool that provides a visual representation of your pipeline configuration. Accessible from the GitLab UI, the Pipeline Editor allows you to interactively modify your .gitlab-ci.yml file directly in the browser.

The Pipeline Editor offers several advantages. Before even commiting changes, it provides real-time feedback on the validity of your YAML syntax and helps prevent common errors. Additionally, it assists in exploring available keywords and includes inline documentation for GitLab CI/CD features. This visual approach can significantly streamline the process of making and validating changes to your pipeline configuration, reducing the need for multiple iterations.

The main limitation is that it can only edit the main .gitlab-ci.yml file.

humanoid orange fox as a mechanic repairing an orange car, lora:DieselPunkAI:.8 DieselPunkAI garage

3. Use gitlab-ci-local to run pipelines locally

Imagine being able to run your pipelines locally without the need to push, wait, or pollute the GitLab instance. Even better, what if you could achieve this without even interacting with the GitLab instance at all?

Initially, the gitlab-runner exec command was intended to fulfill this purpose. However, over time, the code has diverged too much from an actual pipeline running, rendering it capable of handling only 20% of the keywords. As a result, it is no longer usable. There is an official issue on this matter, but progress has been minimal.

Fortunately, the open-source community has made remarkable efforts to implement a local runner, resulting in several options:

GitHub logo firecow / gitlab-ci-local

Tired of pushing to test your .gitlab-ci.yml?

Tired of pushing to test your .gitlab-ci.yml?

Run gitlab pipelines locally as shell executor or docker executor.

Get rid of all those dev specific shell scripts and make files.

build Known Vulnerabilities npm license Renovate

Quality Gate Status Maintainability Rating Reliability Rating Security Rating

Coverage Code Smells Duplicated Lines (%)

Table of contents


Linux based on Debian

Users of Debian-based distributions should prefer the the Deb822 format, installed with:

sudo wget -O /etc/apt/sources.list.d/gitlab-ci-local.sources
sudo apt-get update
sudo apt-get install gitlab-ci-local
Enter fullscreen mode Exit fullscreen mode

If your distribution does not support this, you can run these commands:

curl -s "" | sudo apt-key add -
echo "deb ./" | sudo tee /etc/apt/sources.list.d/gitlab-ci-local.list
sudo apt-get update
sudo apt-get install gitlab-ci-local
Enter fullscreen mode Exit fullscreen mode

Note that the path /etc/apt/sources.list.d/gitlab-ci-local.list is used in the file…

GitHub logo mdubourg001 / glci

🦊 Test your Gitlab CI Pipelines changes locally using Docker.

glci 🦊

Ease GitLab CI Pipelines set-up by running your jobs locally in Docker containers.

Why ? Because I did not want to commit, push, and wait for my jobs to run on the GitLab UI to figure I forgot to install make before running make build.

📣 Disclaimer: this is a helper tool aiming to facilite the process of setting up GitLab CI Pipelines. glci does NOT aim to replace any other tool.


You need to have Docker installed and running to use glci.

yarn global add glci
Enter fullscreen mode Exit fullscreen mode


At the root of your project (where your .gitlab-ci.yml is):

Enter fullscreen mode Exit fullscreen mode

⚠️ You might want to add .glci to your .gitignore file to prevent committing it.


--only-jobs [jobs]

Limiting the jobs to run to the comma-separated list of jobs name given. Handy when setting up that stage-three job depending on that first-stage job artifacts.


glci --only-jobs=install,test:e2e
Enter fullscreen mode Exit fullscreen mode

Adrian DC / gitlabci-local · GitLab

Launch .gitlab-ci.yml jobs locally -


cunity / gitlab-emulator · GitLab

Run most gitlab jobs locally before pushing to your gitlab server. Ideal to validate your yaml changes or test simple builds out locally.


Among these options, the one that has gained the most traction is gitlab-ci-local:

Here are some key features of gitlab-ci-local:

  • The only prerequisite is having functional Docker commands.
  • You can trigger the execution of the entire pipeline, a single job, or the fastest path to a job using the needs: chain.
  • No need to push changes or establish a connection to the server. Committing is also not required.
  • It supports 99% of the keywords used in GitLab CI pipelines and continuously closes the gap. This means that even complex real-life pipelines with artifacts and lightning-fast cache can be executed seamlessly.
  • YAML from includes are seamlessly downloaded
  • GitLab CI predefined variables are populated with valid values.
  • Global, group, and project variables can be managed using files.
  • It supports running unlimited jobs in parallel.

Validate pipeline YAML

To validate your pipeline, simple try to display the jobs list:

$ gitlab-ci-local --list

parsing and downloads finished in 178 ms
name                                   description  stage              when        allow_failure  needs
📦✅-webapp-package-and-test                        📦 package ✅ test  on_success  false
📦✅-backend-package-and-test                       📦 package ✅ test  on_success  false
🐳🧪-webapp-build-push-dev                          🐳 build-push      on_success  false
🐳🧪-backend-build-push-dev                         🐳 build-push      on_success  false
🗑✨-delete-namespace-mr                            ☸ deploy           manual      true
☸🧪-webapp-deploy-dev                               ☸ deploy           on_success  false
☸🧪-backend-deploy-dev                              ☸ deploy           on_success  false
✔-e2e-tests                                         ✔ e2e tests       on_success  false
Enter fullscreen mode Exit fullscreen mode

It helps test your rules. For this, you can also force some predefined variables, to test alternative branching situations.

Run a full pipeline

By default gitlab-ci-local will run the full pipeline for the current branch, or for the current merge request, depending of the predefined variables you overrode. It will trigger and run jobs in parallel, as a GitLab runner would do:

$ gitlab-ci-local
Enter fullscreen mode Exit fullscreen mode

gitlab-ci-local example

Run just a job

We can also run a single job:

$ gitlab-ci-local 📦✅-webapp-package-and-test
Enter fullscreen mode Exit fullscreen mode

Or a job and its dependencies before it (needs: chain and previous stages):

$ gitlab-ci-local 🐳🧪-webapp-build-push-dev --needs
📦✅-webapp-package-and-test           starting node:18.11-alpine (📦 package ✅ test)
📦✅-webapp-package-and-test           copied to docker volumes in 7.72 s
📦✅-webapp-package-and-test           mounting cache for path projects/webapp/node_modules
📦✅-webapp-package-and-test           mounting cache for path projects/webapp/.next/cache
📦✅-webapp-package-and-test           $ cd projects/$MODULE
📦✅-webapp-package-and-test           $ if [ ! -d node_modules/.bin ]; then yarn install; fi
Enter fullscreen mode Exit fullscreen mode

Handle UI variables

Global, group and project variables can be handled in $HOME/.gitlab-ci-local/variables.yml, or locally in each project.

Official example:


    # Will be type Variable and only available if remote is exact match
    AUTHORIZATION_PASSWORD: djwqiod910321 # another syntax

    # Will be type Variable and only available for remotes that include group named 'test-group'
    DOCKER_LOGIN_PASSWORD: dij3213n123n12in3

  # Will be type File, because value is a file path
  KNOWN_HOSTS: "~/.ssh/known_hosts"
    type: variable # Optional and defaults to variable
      "*production*": "Im production only value"
      "staging": "Im staging only value"
    type: file
      "*": |
        Im staging only value
        I'm great for certs n' stuff
Enter fullscreen mode Exit fullscreen mode

humanoid orange fox as a mechanic repairing an orange car, lora:DieselPunkAI:.8 DieselPunkAI garage

4. Declare test branches to simulate long-lived branches and tags

Now that we have the tools to test our pipeline changes with confidence before pushing them, there is still one tricky aspect to consider: how will our pipeline behave on long-lived branches after the merge request is closed?

While we can use gitlab-ci-local locally to force certain variables, such as setting $CI_COMMIT_BRANCH to main in $HOME/.gitlab-ci-local/variables.yml, this approach is only applicable locally and does not serve as valid proof for people reviewing our merge request.

A satisfying solution is to always declare special branch/tag cases and include a corresponding test version. This means that our jobs will run on both main and main-test, on tags and tags-test branch, for example:

    - .staging
    - .deploy-module
    - if: $CI_COMMIT_TAG || $CI_COMMIT_BRANCH == "tag-test"
Enter fullscreen mode Exit fullscreen mode

By doing this, we can test all branches conveniently from our current merge request without actually merging anything. We can create these test branches from our latest commit in the GitLab UI. Once we are satisfied with the pipeline execution (or just the composition of pipeline jobs for sensitive tasks), we can stop the associated pipeline and delete the test branch. If we are not satisfied, we can simply add more commits to our merge request and recreate (or rebase) our test branch(es). When everything seems fine in all scenarios, we can confidently proceed with merging our merge request, knowing that it won't cause any unexpected issues.

Wrapping up

By following these practices and leveraging the available tools, you can significantly reduce the time spent on iterating and testing GitLab CI pipelines. This will lead to a more efficient and enjoyable pipeline development process, ultimately improving your overall development workflow.

So go ahead and implement these tips in your GitLab CI pipeline development, and experience the benefits of faster iterations and more reliable pipelines. Happy coding!

Did this pieces of advice help you ? Do you have others on the same subject ? Please share in the comments below 🤓.

humanoid orange fox as a mechanic repairing an orange car, lora:DieselPunkAI:.8 DieselPunkAI garage

Illustrations generated locally by Automatic1111 using ZavyComics model with DieselPunkAI LoRA

Further reading

This article was enhanced with the assistance of an AI language model to ensure clarity and accuracy in the content, as English is not my native language.

Top comments (2)

fredlight profile image
Frederic Leger

Top ! Constat que je partage au quotidien (cf mon talk au devops d-day de 2023).
Je vais (re)-tester certains de ces outils.

Je pourrais rajouter utiliser un moteur de CI indépendant de votre CI. Les options :
a) tout faire avec make et docker
b) dagger
c) earthly

bcouetil profile image
Benoit COUETIL 💫 • Edited

Le problème avec l'outillage intermédiaire supplémentaire, c'est l'effet boîte noire pour les développeurs, qui viennent dire ensuite "Ta CI ne fonctionne pas" (pas de ownership).

Je préfère considérer que Maven ou NPM sont déjà des makes ! Et la CI est jouable en local avec gitlab-ci-local, comme un make.

Les autres sont aussi des abstractions supplémentaires.

On rentre dans le subjectif là je pense 😉