Introduction
GitLab is a second most-popular DevOps platform that provides a complete solution for source version control, project management, issue tracking, and powerful CI/CD system.
GitLab is constantly working to extend and improve CI/CD functionality, and provides comprehensive documentation for all the new features.
I've been using GitLab for over a decade now for professional and personal projects. In this article I'd like to share some of my findings about useful features and workarounds for some limitations I know of.
Features
Referencing Scripts from Other Jobs
When writing CI/CD pipelines sometimes it is handy to reuse commands between different jobs. GitLab features the !reference custom YAML tag to accomplish this. Below commands from script of .configure-and-build are reused in build-project and build-sub-project jobs.
.configure-and-build:
script:
- cmake -G Xcode -B build
- cmake --build build
build-project:
script:
- !reference [.configure-and-build, script]
build-sub-project:
script:
- cd sub-project
- !reference [.configure-and-build, script]
Using Job Scripts as Functions
The above approach can be extended to use variables as "arguments" to referenced script commands. In below example GENERATOR and BUILD_DIR variables are used to customize behavior of .configure-and-build script. In build-project-macos job .configure-and-build script is invoked multiple times with different GENERATOR and BUILD_DIR values.
.configure-and-build:
script:
- cmake -G ${GENERATOR} -B build
- cmake --build ${BUILD_DIR}
build-project-windows:
...
variables:
GENERATOR: "Visual Studio 16 2019"
BUILD_DIR: "build"
script:
- !reference [.configure-and-build, script]
build-project-macos:
...
script:
- export GENERATOR="Xcode"
- export BUILD_DIR="build_xcode"
- !reference [.configure-and-build, script]
- export GENERATOR="Ninja"
- export BUILD_DIR="build_ninja"
- !reference [.configure-and-build, script]
- export GENERATOR="Unix Makefiles"
- export BUILD_DIR="build_make"
- !reference [.configure-and-build, script]
Multi-line Command Invocation
This is not strictly GitLab CI/CD feature, but rather feature of yaml language which is used to describe CI/CD pipelines in GitLab.
Sometimes it is desired to split long command into multiple lines for readability, but unfortunately bash and powershell use different characters to split multiline command - \ and ` respectively. Therefore, it is not possible to re-use commands for both platforms.
Consider following example, where both windows and macos build jobs have to be duplicated:
build-macos:
...
script: |
cmake -G Xcode \
-B build \
-DOPTION1=YES \
-DANOTHER_OPTION=Test \
-DYET_ANOTHER_OPTION=1 \
-DAND_ANOTHER_OPTION=ON
build-windows:
...
script: |
cmake -G "Visual Studio 16 2019" `
-B build `
-DOPTION1=YES `
-DANOTHER_OPTION=Test `
-DYET_ANOTHER_OPTION=1 `
-DAND_ANOTHER_OPTION=ON
To reduce duplication a "Folded Block Scalar" > can be used to folds newlines to spaces, which unlocks the ability to write multi-line commands that are compatible with both bash and powershell.
.build:
...
script:
- >
cmake -G "${GENERATOR}"
-B build
-DOPTION1=YES
-DANOTHER_OPTION=Test
-DYET_ANOTHER_OPTION=1
-DAND_ANOTHER_OPTION=ON
build-macos:
...
extends:
- .build
variables:
GENERATOR: "Xcode"
build-windows:
...
extends:
- .build
variables:
GENERATOR: "Visual Studio 16 2019"
Exclusive Use of Runners
Sometimes it is desirable to restrict concurrent use of specific runner for a specific job. For example, you might have a runner that can execute unit-tests concurrently, but due to limitations of the underlying framework can only run one instance of end-to-end test application.
Limiting the number of concurrent jobs for the whole runner is suboptimal, since unit-tests can safely be executed in parallel. To resolve this GitLab resource_group keyword can be used to disable execution of a specific job from different pipelines concurrently.
Depending on runner concurrency configuration, multiple unit-test instances can be executed in parallel, but only one instance of test-e2e will be executed concurrently due to presence of resource_group key.
unit-test:
...
tags:
- test-machine-tag
script:
- echo "running unit-tests non-exclusively"
- sleep 60
test-e2e:
...
tags:
- test-machine-tag
resource_group: "test-e2e"
script:
- echo "running e2e tests exclusively"
- sleep 60
Customizing Job Execution Using Rules
GitLab provides powerful rules keyword that is used to customize "when" job is run. Additionally rules supports specifying value for variables on specific conditions that can help to customize "how" job is being executed.
Consider the example below where variable SIGN will be set to YES only when build job is executed for tagged commit, in other cases it will be NO.
build-macos:
variables:
SIGN: "NO"
rules:
- if: "$CI_COMMIT_TAG"
variables:
SIGN: "YES"
- when: always
script: |
echo "Executables will be signed: ${SIGN}"
Clearing Working Directory for Specific Job
Recently GitLab added option to clean build directory before downloading artifacts and executing job scripts. To achieve this, GIT_STRATEGY variable should be set to empty. Per GitLab documentation:
Use the
emptyGit strategy when:
- You do not need the repository data to be present.
- You want a clean, controlled, or customized starting state every time a job runs.
Example:
job:
...
variables:
GIT_STRATEGY: empty
script:
- ls -al
Workarounds
Spaces in Variables
Consider a scenario where you want to have a variable that has list of file paths to be passed to some script and file paths may contain spaces. Example below contains list of files to be passed to signing script:
job:
variables:
FILES_FOR_SIGNING: "'Test App 1/Test App 1' 'Test App 2/Test App 2'"
script:
- echo $FILES_FOR_SIGNING
- ./sign.sh $FILES_FOR_SIGNING
sign.sh script:
#!/bin/sh
if [ $# -eq 0 ]; then
echo "No arguments provided"
exit 1
fi
i=1
for arg in "$@"; do
echo "$i: $arg"
i=$((i + 1))
done
Output of pipeline execution will look like below - with arguments split by spaces, without quotes being taken into account.
$ echo $FILES_FOR_SIGNING
'Test App 1/Test App 1' 'Test App 2/Test App 2'
$ ./sign.sh $FILES_FOR_SIGNING
1: 'Test
2: App
3: 1/Test
4: App
5: 1'
6: 'Test
7: App
8: 2/Test
9: App
10: 2'
I have tried different combinations of quotes and escaped quotes, but the result was the same all the time. But, if you have noticed, echo $FILES_FOR_SIGNING produces string with single quotes preserved. Combining it with eval solves the problem!
Replacing ./sign.sh $FILES_FOR_SIGNING with eval $(echo ./sign.sh $FILES_FOR_SIGNING) solved the issue and the output was:
$ eval $(echo ./sign.sh $FILES_FOR_SIGNING)
1: Test App 1/Test App 1
2: Test App 2/Test App 2
A note on security: using eval in certain conditions is considered unsafe, even with current example FILES_FOR_SIGNING can be easily constructed so that it can execute arbitrary commands, use with caution!
Colon+Space in Values
I have stumbled upon this issue several times and each time it took me a while to figure out what is wrong. In short - if you want to use colon and space : in the value, whole value string must be quoted.
Obviously, this will work fine:
job:
variables:
ANIMALS: "Animals: dog, cat, mouse"
script:
- echo $ANIMALS
But this will not, as it is not compliant with above rule:
job:
script:
- echo "Animals: dog, cat, mouse"
To make it compliant, whole line should be quoted, no matter it is dictionary or a list value, like this:
job:
script:
- 'echo "Animals: dog, cat, mouse"'
Conclusion
GitLab is a powerful and extensive tool that offers a wide range of features for creating advanced CI/CD pipelines. Throughout this article, we've explored several techniques and workarounds that can help you create more efficient and flexible pipelines, streamline your development process and accelerate your software delivery.
Remember that GitLab is constantly evolving, so it's worth staying up-to-date with the latest features and best practices.
Top comments (0)