DEV Community

Artem Ptushkin
Artem Ptushkin

Posted on

Handle Gitlab pipeline duplication

#gitlab #devops #cicd #git

Problem

There is a known problem that in Gitlab it's not possible to avoid pipeline duplication in some scenarios.

Context

Let's assume you combine merge and branch pipelines. You can follow the Gitlab guideline or here is an improved one:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_PIPELINE_SOURCE == "web"'
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never
    - if: '$CI_COMMIT_BRANCH'
Enter fullscreen mode Exit fullscreen mode

When you push and then you open a MR immediately it's not possible to avoid getting a second pipeline (though if you have one open then it's not an issue).

Solution with git options 

git push -o merge_request.create
Enter fullscreen mode Exit fullscreen mode

This solution with the script below

Here is a script that supposed to be run after the push to cancel the duplication.

How to use?

  1. Create an alias in your bash profile / .zshrc file, e.g.: alias cancel-branch-pipeline="sh ~/gitlab/gitlab-cancel-branch-pipeline.sh"
  2. Set you Gitlab personal access token with scopes api and read_api as GITLAB_TOKEN environment variable (you can set it in your bash profile file to have in every session: export GITLAB_TOKEN=<your-token>)
  3. Call it after with the alias anytime you push, and you want to open a MR: $ cancel-branch-pipeline

Top comments (8)

Collapse
 
bcouetil profile image
Benoit COUETIL 💫 • • Edited

My friend, the two modes, branch pipeline and MR pipelines, were meant to be triggered on every occasion, depending on variables you use.

By default, there are only branch pipelines, and if you use some variables present in both cases, you may trigger also MR pipeline.

workflow:rules/rules are not meant to be that complicated. When reading yours, it is very difficult to understand in which case it is triggered.

Be more straightforward, even if more verbose, it does not hurt.

Let's say you want :

  • a pipeline on develop and main branches
  • a pipeline on MR targeting develop
  • not on MR targeting main, because it is from develop and there is already a pipeline for each

The workflow rules would be :

workflow:
  rules:
    # OK on MR to develop
    - if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "develop"
    # OK on main branches
    - if: $CI_COMMIT_BRANCH == "develop"
    - if: $CI_COMMIT_BRANCH == "main"
Enter fullscreen mode Exit fullscreen mode
Collapse
 
art_ptushkin profile image
Artem Ptushkin •

Hey, it's deeper, I'm sorry, I know all the basics thanks. There is a known problem or pipeline duplication when you push and then immediately, that's what I wrote

When you push and then you open a MR immediately it's not possible to avoid getting a second pipeline (though if you have one open then it's not an issue).

Collapse
 
art_ptushkin profile image
Artem Ptushkin •

I leverage the switch between branch and MR pipelines, it's supported by Gitlab here, having this you can find the root reason of the issue

Thread Thread
 
bcouetil profile image
Benoit COUETIL 💫 •

Sorry I read your article too quicky, I should have taken the time needed.

What do we gain, switching from one type of pipeline to the other for the same branch ?

Seems interesting but a use case would help me grasp it.

Thread Thread
 
art_ptushkin profile image
Artem Ptushkin •

Np, I didn't pay attention to the reasons either. Thanks for asking.

The switching is for developers who prefer working on a branch and opening an MR when he's really ready, example flow:

  1. you work, commit, and push
  2. you get a pipeline
  3. you make sure it's green
  4. you open the MR when you're ready to show it and you get pipelines there now aka public

The problem happens between 3 and 4 if they happen one after another - simple one-commit merge requests are always ready for review immediately.

Thread Thread
 
bcouetil profile image
Benoit COUETIL 💫 • • Edited

OK, so it is interesting if the pipeline is different ; else keeping the branch pipeline type all the way would suffice (no MR pipeline type, but still branch pipelines from now on ; the default behavior).

Do you have examples of differences in your pipelines, before and after the MR creation ? E2E tests added maybe ?

Thread Thread
 
art_ptushkin profile image
Artem Ptushkin •

No, no difference I have. You mostly need a merge requests pipeline to run pipelines from the merge request web page

Image description

You're right branch one is linked by default to each MR but you can not run that button. That's how we ended up combining solutions in one pipeline code, we provide one common pipeline for all.

Thread Thread
 
bcouetil profile image
Benoit COUETIL 💫 •

So much for a shortcut button 😅.

But it has opened me possibilities.

Thanks !

Read next

habdulr1 profile image

Linux File System

habdulr1 -

smythp profile image

Deep Dive 🤿: Where Does Grype Data Come From?

Patrick Smyth -

luqmanbello profile image

Building Production-Grade ECS Anywhere Infrastructure with Custom Capacity Providers

Luqman Bello -

evgenyigumnov profile image

Need Advice on Improving My Codebase: Increased Readability and Error Management

Evgeny Igumnov -