An excellent way to deal with (lerna) monorepos in CircleCI: circletron

We started with a monorepo at my current company and have been using circle almost since the beginning. It was tough. It required a lot of boilerplate in our config and the necessity to get every developer to generate a circle API key and add it to git. It never felt that great but at least it worked. In April CircleCI released the dynamic configuration API and this allowed us to refactor our monorepo support into something we think is pretty great. We've gone from tolerating Circle to enjoying it, and now we've released the code as an open source project and provided the functionality as an orb so that any project can benefit.

The project is circletron, the code is hosted on github and the orb is here. It currently supports lerna monorepos but we plan to support other monorepos in the future.

Motivating example

Typically a circleci configuration exists in a single file within a repository at .circleci/config.yml. For a monorepo a very minimal example may look something like this:

version: 2.1

jobs:
  validate-everything:
    steps:
      - checkout
      - run: npm run general-validation

  test-subpackage-a:
    steps:
      - checkout
      - run: cd packages/package-a && npm run test

  test-subpackage-b:
    steps:
      - checkout
      - run: cd packages/package-b && npm run test

  test-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-c && npm run test

  publish-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-c && npm run publish


workflows:
  validate-everything:
    jobs:
      - validate-everything

  subpackage-a:
    jobs:
      - test-subpackage-a

  subpackage-b:
    jobs:
      - test-subpackage-b

  subpackage-c:
    jobs:
      - test-subpackage-c
      - publish-subpackage-c

Obviously there are many problems with this, for a start all the CI is defined in one file at the root of the project. Worse, each of the jobs may take some time to complete and on every commit to the project, circleci will run every single job no matter whether there are any changes to the respective subpackages or not. It may not look so bad in this tiny example but the bigger the configuration gets, the worse it is to deal with.

Using circletron to solve this

With circletron the .circle/config.yml is always the same:

version: 2.1
setup: true
orbs:
  circletron: circletron/circletron@3.0.1

workflows:
  trigger-jobs:
    jobs:
      - circletron/trigger-jobs

The trigger job step will take many individual circle.yml distributed within the project and combine them into a single configuration which will be issued via the continuation API. It will modify the configuration to ensure that jobs that are not necessary are no longer run, in a way that is friendly to CI branch protection rules.

The single configuration file can now be split up across the monorepo, with one optional circle.yml in the root of the project and where the CI for each subpackage lives in the directory for that subpackage.

In this instance the circle.yml in the root of the project will host configuration not specific to any subpackage:

version: 2.1

jobs:
  validate-everything:
    steps:
      - checkout
      - run: npm run general-validation

workflows:
  validate-everything:
    jobs:
      - validate-everything

The jobs here are run on every commit and it's also a good place to set version. It's also a good place to provide commands that can be used in subpackage specific circle.yml files.

Now lets look at packages/package-a/circle.yml:

jobs:
  test-subpackage-a:
    steps:
      - checkout
      - run: cd packages/package-a && npm run test

workflows:
  subpackage-a:
    jobs:
      - test-subpackage-a

Everything related to this package all in one place. Even better, when the PR contains no changes within packages/package-a the jobs for this subpackage will be skipped. You will probably want to use branch protection rules to ensure that when the test-subpackage-a job fails the PR will be blocked so omitting this job entirely would not be ideal. Omitted jobs remain in pending state permanently, blocking the PR. circletron replaces unneeded jobs with a simple job using the busybox:stable docker image that echoes "Job not required" and issues a success error code.

Dependencies and unconditional jobs

What if the code in packages/subpackage-c uses code from packages/subpackage-a? In this case the jobs within this package should run when the code in subpackage-a changes, even if the code in the subpackage itself doesn't change.

There may also be some instances where a job should run on every push.

The configuration at packages/subpackage-c/circle.yml shows how to add dependencies and create jobs which run unconditionally:

dependencies:
  - subpackage-a

jobs:
  test-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-b && npm run test

  publish-subpackage-c:
    conditional: false
    steps:
      - checkout
      - run: cd packages/package-c && npm run publish

workflows:
  subpackage-c:
    jobs:
      - test-subpackage-c
      - publish-subpackage-a

In this instance test-subpackage-c will only be run when changes to package-a or package-c are detected and publish-subpackage-c will run on every push.

Target branches

CircleCI does not pass the target branch to workflows or jobs so it becomes necessary to help circletron out. By default circletron considers the branches main, master, develop and any branch starting with release/ as a target branch. The latest commit from the branch commit history which belongs to one of these branches is considered to be the branch-point. For pushes to one of the branches above, all of the jobs are run. This can be changed via the configuration file at .circleci/circletron.yml:

targetBranches: ^(release/|main$|master$|develop$)

Any branch which matches this regex is considered a target branch, the above configuration shows the default.

circletron is available to use now.