DEV Community

Cover image for Let CI check & fix your yamls
adriens for opt-nc

Posted on • Updated on

Let CI check & fix your yamls

πŸ‘‰ Intro

A few months ago, we have started to migrate our sourcecode CI & CD to Github.com.

Then, both DEV and OPS started to migrate source codes.

It appeared that OPS had to migrate a huge amount of Ansible related repos from an onPrem Gitlab instance to Github.com, and we wanted to take advantage of linting and code quality of the yamls.

OPS started to develop the migration process so it could be entirely code driven and tested.

What appeared was that we needed to manage that huge amount of code to :

  • 1️⃣ lint (syntax validity, but for weirdnesses like key repetition and cosmetic problems such as lines length, trailing spaces, indentation, etc)
  • 2️⃣ fix/implement lints feedbacks in a massive way (they were so huge that we could not imagine do that by hand)

OPS did choose to use the following approach :

  • 1️⃣ Lint with yamllint
  • 2️⃣ Fix with yamlfixer, which in facts uses yamllint in his core

As DEV & OPS are using yaml everyday a -bit- lot more (k8s, CI configuration, Spring Boot configuration, ...), we started to

think about a common toolbox that we could embed in our CI pipeline, on Github.com.

What we needed to do was to :

  1. Integrate OPS sourcecode (DevOPS teamwork)
  2. Package it inside as a dedicated GH Action that everyone could add to their pipeline within a few lines of code

πŸ› οΈ Yaml Fixer

An important work has been done by OPS on yamlfixer so it could be used conveniently from the cli :

GitHub logo opt-nc / yamlfixer

Automates the fixing of problems reported by yamllint by parsing its output

License Build

Docker Image

PyPI PyPI - Python Version PyPI - Downloads

yamlfixer's logo yamlfixer

❔ About

yamlfixer automates the fixing of problems reported by yamllint by parsing its output.

πŸ–ΌοΈ Screenshot

Here'a screenshot of yamlfixer launched on yaml-test-suite :

yamlfixer's screenshot

πŸ“‘ Installation

The easiest way to install yamlfixer is from pypi, as described below.

🐧 Linux install

python3 -m pip install yamlfixer-opt-nc
Enter fullscreen mode Exit fullscreen mode

πŸͺŸ Windows install

python -m pip install yamlfixer-opt-nc
Enter fullscreen mode Exit fullscreen mode

pipx setup

For an optimal experience we recommand using pipx.

To install :

pipx install yamlfixer-opt-nc
pipx list
Enter fullscreen mode Exit fullscreen mode

To upgrade :

pipx upgrade yamlfixer-opt-nc
Enter fullscreen mode Exit fullscreen mode

To uninstall :

pipx uninstall yamlfixer-opt-nc

πŸš€ Usage

This software automatically fixes some errors and warnings reported by yamllint.

usage: yamlfixer [-h] [-v] [-b] [-B BACKUPSUFFIX] [-d] [-D DIFF_FILE] [-e EXTENSIONS] [-f]
                 [-F] [-l] [-N] [-n] [-r LEVEL] [-j | -p | -s] [-t TABSIZE]
                 [-c CONFIG_FILE | -C CONFIG_DATA]
                 [FILE_or_DIR [FILE_or_DIR ...]]
Fix formatting problems in YAML documents. If no file is specified, then reads input
…
Enter fullscreen mode Exit fullscreen mode

Still this cli needed a Python runtime to be used.

🐳 Deliver as a Docker image

Then we wanted to distribute it an easier way, which meaned :

package it as a Docker image.

At that point, DEVs took the relay and implemented Docker release on GH CI, and pushed images on DockerHub under optnc/yamlfixer :
Image description

πŸƒ Deliver as a GH Action

Now, we could use the Docker image to implement the following Public GH Action :

GitHub logo opt-nc / yamlfixer-action

Github action that fixes some errors and warnings reported by yamllint and create a pull request with fixes.

❔ About

This github-action is based on the tool Yamlfixer :

It automatically fixes some errors and warnings reported by yamllint and creates a pull request that embeds the fixes.

🧰 Usage

See action.yml

You need a Github account that is allowed to create pull request on the inspected repository.

Find below an example that allows to check yaml files on each push with verbose mode :

name: Lint yaml files

on: [push]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout my app
        uses: actions/checkout@v3
      - name: Lint yaml files
        uses: opt-nc/yamlfixer-action
        with:
            options: --summary
            user: ${{secrets.my_user}}
            token: ${{secrets.my_user_password}}
Enter fullscreen mode Exit fullscreen mode

The github-action creates :

  1. A new branch named yamlfixer/patch/$branch_name
  2. The pull request to be merged into the working branch

πŸ”– Resources

Here are some useful resources…

... that could be released on the Marketplace :

Image description

πŸ€– Action usage

Here is the code (and... it's yaml πŸ”πŸ₯šπŸ£) required to use the action in any workflow :

name: Lint yaml files

on: [push]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout my app
        uses: actions/checkout@v2
      - name: Lint yaml files
        uses: opt-nc/yamlfixer-action
        with:
            yaml_file: .github/*.yml
            options: --verbose
            user: ${{secrets.my_user}}
            token: ${{secrets.my_user_password}}
Enter fullscreen mode Exit fullscreen mode

πŸ‘‰ Notice that the action creates a new branch and the pull request to be merged into the working branch so this code activity can be managed and reviewed as any other source code proposal.

🎦 Live demo

As a live video worths a thousand words : discover how the Github Action runs on a dedicated repository :

πŸ“· Examples in details

❌ Rejected PR because of lint failure

Image description

βœ”οΈ Valid PR coming from GH Action

Image description

And code change review :

Image description

πŸͺ™ ROI

Now, both DEVs and OPs are using it to enhance yaml quality proactively with the help of CI and yaml quality issues are managed on our daily workflow on GH Issue.

πŸ‘‰ Also notice that all the teams using the GH Action have the same level of quality accross the whole organization.

πŸ™ Acknowledgments πŸ§‘β€πŸ€β€πŸ§‘

A lot of thanks to the team that made this work come true, all on our DEVOPS pipeline :

  • Jerome ALET as core Yaml Fixer developer & integrator
  • MichΓ¨le BARRE for Docker packaging and GH Action development
  • Daniel SANTOS for its test on DockerHub publish and his continuous feedbacks on this new Gh Action

Discussion (0)