How to configure CircleCI: what you might not know

CircleCI is not the most popular CI/CD platform for open source contributors today, but it is widely used by many big sharks in the industry. So we have to keep in mind how to accurately configure it.

CircleCI offers several execution environments. Those bold nerds from San Francisco call them executors. An executor defines the underlying technology or environment in which to run a job. Set up your jobs to run in the docker, machine, macos or windows executor and specify an image with the tools and packages you need.

Also, CircleCI provide using so-called orbs with its platform. Orbs are open-source, shareable packages of parametrizable reusable configuration elements, including jobs, commands, and executors. You may involve orbs to reduce configuration complexity and help you integrate with your software and services stack quickly and easily across many projects.

In general, orb concepts are similar to GitHub Actions Marketplace items' ones. This feature enables snippets of code to be re-used to automate repetitive tasks and build lightweight pipelines as code using YAML.

A top hierarchy unit in CircleCI flow is called workflow. A workflow divided to jobs. A job may include many steps.

If you use orbs to perform some standard actions, for example, to build and publish a docker image, then the system works strictly according to the template. Unfortunately, the documentation is sometimes complicated and confusing. One may get the impression, as happened in my case, that the orbs live their own lives, their behaviour cannot be changed. For example, I needed to achieve activation of docker layer caching (DLC) when building images, so as not to re-use previous layers and thereby save resources and time.

⚠️💰Note: The so-called Performance plan is required to use Docker Layer Caching. If you are on the container-based plan you will need to upgrade the plan.

Basically, a layer, or image layer is a change on an image, or an intermediate image. Every command you specify (FROM, RUN, COPY, etc.) in your Dockerfile causes the previous image to change, thus creating a new layer. You can think of it as staging changes when you're using git: you add a file's change, then another one, then another one... Re-using unchanged data in the build is what we need.

As it turned out, it is impossible to use a custom docker environment or a setup_remote_docker parameter with orbs, but you can still enable DLC and achieve a win up to 35% in flow runtime! I'll explain below how to do the trick.

There is an option to tweak an executor globally at the start of your flow. CircleCI doesn't allow splitting flow files, so it will be a common setting.

I. First let's describe our custom executor which we'll be able to re-use later in our job:

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@7.2.0
executors:
  my-customized-executor:
    machine:
      docker_layer_caching: true
      image: 'ubuntu-2004:202101-01'
    resource_class: medium

Almost everything in CircleCI runs upon docker, so, as you may see we just turned on DLC for that runner. Do not try to imagine how it works, but trust me, it works.

II. Now we may point CircleCI to use our customized machine somewhere we need among our jobs:

jobs:
  build-and-push-image:
    executor: my-customized-executor
    steps: ...

III. Launch your flow. Read the logs carefully. With DLC enabled you should see from docker:

...
Step 2/8 : RUN apt-get update && apt-get install -y build-essential && apt-get install -y python
 ---> Using cache
...

Here it is!

Not all solutions lie on the surface, but by swimming to the iceberg, without being frightened and having consumed it, you can get, e.g., fresh drinking water.

We at Otomato wish our colleagues, let the best solutions be found quickly even when it comes to complex and insidious CircleCI. 😎