"to be continuous": opinionated GitLab CI

#gitlabci #devops #opensource

Introduction

At Orange, we have been using GitLab for several years (since late 2015).
By the way we have several self-managed GitLab servers. Some in Community Edition, others in Enterprise Edition, but the overwhelming majority of developers at Orange manage their code in a GitLab.

GitLab is a fantastic tool, promoting a very modern and ambitious vision of development (collaboration, Merge Requests, DevOps, continuous integration …).
The monthly releases of GitLab are each time a gift that we cannot wait to unwrap to discover the many new toys that the fox firm has prepared for us.

At Orange, for CI/CD too, we've smoothly moved from Jenkins to GitLab CI.
For my part, I started using GitLab CI in 2017 and I was immediately struck by the simplicity and versatility of its architecture (1 job = 1 container, no API to implement, no nightmare upgrading the product and its plugins, decoupling of runners, …). I immediately adopted it and never came back to Jenkins again.
As a result, after 2 years of practice I had acquired some savoir-faire, and I was able to start sharing my expertise and some pieces of GitLab CI code.

Then in January 2019, the include feature was released in the Community Edition (version 11.7.0). It was truly an epiphany.
We could now go much further than just sharing pieces of code between colleagues: we could share ready to use templates…

At that time, I had just got a position of Developer Advocate at Orange, and supporting Orange developers by providing state-of-the-art GitLab CI templates seemed to be perfectly inline with the job description.
I was also an active member of the "Orange Experts: Software" community, it was only natural that I asked some of my fellow colleagues to work together on this topic.

During 2 years, we developed templates, we polished them, we increased our GitLab CI expertise, we also progressed in our semantic representation of what a pipeline should be, the nature of the tasks, the profound differences between CI and CD, Git workflows, …

From the very beginning we designed these templates as a honeypot. On the one hand, an argument to attract newbies who wanted to set up their continuous integration without spending hours on GitLab CI, natively implementing all the killer-features offered by GitLab, and on the other hand a way to accelerate/enforce the adoption of state-of-the-art tools and practices (DevSecOps tools, modern and consistent workflows, speed and frugality of CI pipelines, sureness and completeness of CD pipelines).

In only 2 years, this personal initiative became a recognized inner source project, attracting more and more users and contributors, bringing the community to nearly 1000 members today (June 2021), and about 30 templates available.
The visible effect of this increasing adoption is the beginning of a CI/CD standardization at Orange.

Orange - like many IT companies - is a heavy consumer of opensource, sharing in return what we do well is just obvious.
The logical continuation of this story was going opensource, which happened in May 2021 under the name "to be continuous".

https://to-be-continuous.gitlab.io/doc

To semantics and beyond …

What does to be continuous bring?

I am a big fan of GitLab and GitLab CI. But one shall admit that building your DevOps pipeline from scratch requires a lot of skills and time if you want to take advantage of the many features.

In addition, GitLab CI is a product that is constantly evolving, and each release brings a lot of new things. Integrating these new features over time can represent a significant recurring burden for projects.

And finally, the most problematic point in my opinion is the lack of semantic framework provided by GitLab CI. GitLab CI is mainly a sophisticated scheduler, allowing to define technical tasks and their sequence, but GitLab CI cares little about the nature of these tasks, and does not give any clues as to the "right" way to build a DevOps pipeline. The consequence of this is a lack of standardization from one project to another, from one company to another. This is a criticism that can be formulated against the majority of CI/CD tools today.
I am quite convinced that this will be the next big topic in the CI/CD products market.

In fact, as a veteran Javaist (fancy phrase to admit that I'm not that young anymore), the current situation reminds me of what was the Java build in the pre-Maven era. Back then, we used non-structuring tools such as Make or Apache Ant, each project built up its own build system, adopted its own conventions, its own code and resource files structure, … in short, it was a happy mess, everyone reinvented the wheel, and when joining another project, it was necessary to relearn "how does the build work here?".
Then in 2004, Maven was released (and Gradle 3 years later). For a while there were heated debates between the proponents of standardization and the defenders of expertise and of "I can control how it works", but today it would not occur to anyone to build a Java project other than with Maven or Gradle. If I join a project developed in Java, I will immediately know how files are organized, how the project is built. To summarize: the Java build has become a non-topic.

I think this is what must happen now in the field of CI/CD: tools shall offer a more opinionated framework so that the CI/CD in turn becomes a non-topic .

In short, this is what to be continuous modestly aims: to lay the foundations for a standardization of the CI/CD through semantic modeling, integrating Git workflows, environment management, the nature of the tasks of a pipeline, the differences between CI and CD …

Like Maven or Gradle, to be continuous comes with conventions (default configuration), but remains open to customization nonetheless. Either by overloading configuration variables, or by overwriting YAML code.

Why opensourcing?

GitLab is a widely used product; by many companies and developers (around 2/3 of the market share of on-premise CI/CD products).

At Orange, we needed to develop these templates to start standardizing our CI/CD and systematizing the adoption of state-of-the-art tools and practices. We believe this is a general need.

In addition, DevSecOps tools (static code analysis tools, acceptance testing tools, security tools, compliance tools, etc.) are constantly evolving. Integrating new tools and keeping up with developments requires time and expertise that we hope to find in the opensource community.

Opensourcing to be continuous is also the hope of creating a larger community, promoting our vision and sharing development efforts.

What is in "to be continuous"? How can I start?

For now, to be continuous has 26 templates of 6 kinds:

Build & Test: Angular, Bash, Go, Gradle, Maven, MkDocs, Node.js, PHP, Python
Code Analysis: Gitleaks, SonarQube Packaging: Docker
Infrastructure (IaC): Terraform
Deploy & Run: Ansible, Cloud Foundry, Google Cloud, Helm, Kubernetes, OpenShift, S3 (Simple Storage Service)
Acceptance: Cypress, Postman, Puppeteer, Robot Framework, SSL test
Others: semantic-release

To be continuous is thoroughly documented (basic notions and philosophy, general usage principles, every template also has its own documentation, how to use to be continuous in a self-managed GitLab).

To get started quickly, to be continuous provides an interactive configurer (a.k.a "kicker") that allows generating the .gitlab-ci.yml file simply by selecting the technical characteristics of your project.

Finally, to be continuous exposes several example projects, illustrating how to use the templates in production-like projects, combining multiple templates.