Do you want to ship bug-free code at high velocity? A fast and reliable CI/CD pipeline is crucial for doing that sustainably over time.

What is a CI/CD pipeline?

A CI/CD pipeline helps you automate steps in your software delivery process, such as initiating code builds, running automated tests, and deploying to a staging or production environment. Automated pipelines remove manual errors, provide standardized development feedback loops and enable fast product iterations.

What do CI and CD mean?

CI, short for Continuous Integration, is a software development practice in which all developers merge code changes in a central repository multiple times a day. CD stands for Continuous Delivery, which on top of Continuous Integration adds the practice of automating the entire software release process.

With CI, each change in code triggers an automated build-and-test sequence for the given project, providing feedback to the developer(s) who made the change. The entire CI feedback loop should run in less than 10 minutes.

Continuous Delivery includes infrastructure provisioning and deployment, which may be manual and consist of multiple stages. What's important is that all these processes are fully automated, with each run fully logged and visible to the entire team.

Elements of a CI/CD pipeline

A CI/CD pipeline may sound like overhead but it really isn't. It's essentially a runnable specification of the steps that need to be performed in order to deliver a new version of a software product. In the absence of an automated pipeline, engineers would still need to perform these steps manually, and hence far less productively.

Most software releases go through a couple of typical stages:

Failure in each stage typically triggers a notification—via email, Slack, etc.—to let the responsible developers know about the cause. Otherwise notifications are usually configured to be sent to the whole team after each successful deploy to production.

Source stage

In most cases a pipeline run is triggered by a source code repository. A change in code triggers a notification to the CI/CD tool, which runs the corresponding pipeline. Other common triggers include automatically scheduled or user-initiated workflows, as well as results of other pipelines.

Build stage

We combine the source code and its dependencies to build a runnable instance of our product that we can potentially ship to our end users. Programs written in languages such as Java, C/C++, or Go need to be compiled, whereas Ruby, Python and JavaScript programs work without this step.

Regardless of the language, cloud-native software is typically deployed with Docker, in which case this stage of the CI/CD pipeline builds the Docker containers.

Failure to pass the build stage is an indicator of a fundamental problem in the configuration of our project and it's best to address it immediately.

Test stage

In this phase we run automated tests to validate the correctness of our code and the behavior of our product. The test stage acts as a safety net that prevents easily reproducible bugs from reaching the end users.

The responsibility of writing tests falls on the developers, and is best done while we write new code in the process of test- or behavior-driven development.

Depending on the size and complexity of the project, this phase can last from seconds to hours. Many large-scale projects run tests in multiple stages, starting with smoke tests that perform quick sanity checks to end-to-end integration tests that test the entire system from the user's point of view. A large test suite is typically parallelized to reduce run time.

Failure during the test stage exposes problems in code that developers didn't foresee when writing the code. It's essential for this stage to produce feedback to developers quickly, while the problem space is still fresh in their minds and they can maintain the state of flow.

Deploy stages

Once we have a built a runnable instance of our code that has passed all predefined tests, we're ready to deploy it. There are usually multiple deploy environments, for example a "beta" or "staging" environment which is used internally by the product team, and a "production" environment for end users.

Teams that have embraced the Agile model of development—which is guided by tests and real-time monitoring—usually deploy work-in-progress manually to a staging environment for additional manual testing and review, and automatically deploy approved changes from the master branch to production.

Examples of CI/CD pipelines

A pipeline can start very simple. Here's an example of a pipeline for a Go project which compiles the code, checks code style and runs automated tests in two parallel jobs:

The pipeline is implemented with Semaphore, a cloud-based CI/CD service.

Here's a more complex example of a pipeline that builds, tests and deploys a microservice to a Kubernetes cluster:

And here's how to build it:

Semaphore documentation provides more examples of CI/CD pipelines.

Additional benefits of pipelines

Having a CI/CD pipeline has more positive effects than simply making what was previously done a little bit more efficient:

• Developers can stay focused on writing code and monitoring the behavior of the system in production.
• QA and product stakeholders have easy access to the latest, or any, version of the system.
• Product updates are not stressful.
• Logs of all code changes, test and deployments are available for inspection at any time.
• Rolling back to a previous version in the event of a problem is a routine push-button action.
• A fast feedback loop helps build an organizational culture of learning and responsibility.

Happy building!