YAML Buildpack: Auto Validate Configuration Repositories

#yaml #buildpack #gitlab #autodevops

Git is one of the most common buckets to manage systems/services configurations, which is obvious due to the benefits brought with Git (for example, tracking changes and rolling back).

In my company, we use Git to have a remote configuration for our services and tools. There are over 700 configuration repositories 🚀

Keeping in mind that:

Configuration changes are error-prone. Making sure any change in configuration doesn’t break the system is important to avoid critical incidents[1].

So:

As a User, I want to make sure that my changes are in the correct format, so I avoid having system troubles due to wrong configuration files.

CI/CD can be used to apply any kind of checks before merging the change to the *main *branch, however, it comes to a cost to introducing an Action or GitLab-CI (depending on the tool that you use) to make it happen. The big cost here is that you want to update every repository to have the proper CI definitions for that.

YAML Validator

Recently, we migrated from BitBucket to GitLab. In BitBucket, we were using YAML Validator Pre-Receive hook[2] which was doing the job but unfortunately, there is no easy way to have a similar in GitLab.

Alternatives in GitLab

👎 Global server-hook (It is not easy to manage in a Docker container of GitLab Server)
👎 Client side commit hooks (Not practical)
👎 Validation Job through .gitlab-ci.yml to be introduced in every repository (High cost to introduce it for that many configuration repositories, besides requiring config repositories to maintain)
👍 Auto DevOps! (Centralized setup and easy to maintain 😎)

GitLab Auto DevOps

GitLab Auto DevOps[4] was a good candidate to be used to solve the problem as it doesn’t require to do any changes to the configuration repositories.

GitLab Auto DevOps is a collection of pre-configured features and integrations that work together to support your software delivery process.

GitLab DevOps is based on buildpacks[5]:

A Buildpack is a set of executables that inspects your app source code and creates a plan to build and run your application.

The existing Buildpacks don’t have a YAML validator in between, hence the default GitLab DevOps doesn’t fit automatically without some customizations[6]

Auto Test: YAML Validator Buildpack

To use GitLab Auto DevOps — Auto Test[7], a Buildpack to validate the configuration files (which are Yaml based) is required. Here the Yamllint-buildpack[8] was introduced to have this job done 💻

Detection

For the YAML Buildpack to run, It checks first if the repository is matching. If the repository contains YAML files, it prints the Buildpack name, otherwise, it terminates the script.

#!/usr/bin/env bash
set -eo pipefail

ymlFilesArray=$(find "${1:?}" -maxdepth 1 -regex ".*.[yml|yaml]" | grep -v '^$')

if [ ${#ymlFilesArray[@]} -gt 0 ]; then
  echo "freenow/yamllint"
  exit 0
else
  (>&2 echo "Could not find yaml/yml files")
  exit 1
fi

Test compile

This step is to prepare the repository for the test. As there is no compilation required before running the tests, the test-compile script is empty. The file MUST exist for the Herokuish to successfully run.

Test

This script is to apply the YAML validation using yamllint.

To run the yamllint check:

Download and Install Python
Install Pip
Install yamllint
Run yamllint check

For the complete script:

#!/usr/bin/env bash
# bin/test <build-dir> <env-dir>

set -eo pipefail 

echo "---> yamllint check..."   

# GET ARGS
BP_DIR=$(cd $(dirname $0)/..; pwd) 

# absolute path
BUILD_DIR=$1
ENV_DIR=$2  

# CREATE THE LAYER DIRECTORY
pythonlayer="$ENV_DIR"/python
mkdir -p "$pythonlayer" 

# DOWNLOAD PYTHON
echo "---> Downloading and extracting Python"
python_url=https://s3-external-1.amazonaws.com/heroku-buildpack-python/heroku-18/runtimes/python-3.10.0.tar.gzwget -q -O - "$python_url" | tar -xzf - -C "$pythonlayer" 

# MAKE PYTHON AVAILABLE TO THIS SCRIPT
export PATH="$pythonlayer"/bin:$PATH
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}"$pythonlayer/lib" 

# INSTALL PIP
echo "---> Installing pip"
python -m ensurepip --upgrade 

# INSTALL YAMLLINT
echo "---> Installing yamllint"
pip3 install yamllint 

# RUN YAMLLINT
echo "---> Validating yaml/yml files"
yamllint -c "$BP_DIR/config/yamllint-config" $BUILD_DIR 

echo "---> [success] yaml/yml validation"

Additional configurations

GitLab is by default enabling not needed jobs by default. Due to that, you need to disable these jobs by using GitLab variables and set the below variables as false