We started with a monorepo at my current company and have been using circle almost since the beginning. It was tough. It required a lot of boilerplate in our config and the necessity to get every developer to generate a circle API key and add it to git. It never felt that great but at least it worked. In April CircleCI released the dynamic configuration API and this allowed us to refactor our monorepo support into something we think is pretty great. We've gone from tolerating Circle to enjoying it, and now we've released the code as an open source project and provided the functionality as an orb so that any project can benefit.
Typically a circleci configuration exists in a single file within a repository at
.circleci/config.yml. For a monorepo a very minimal example may look something like this:
version: 2.1 jobs: validate-everything: steps: - checkout - run: npm run general-validation test-subpackage-a: steps: - checkout - run: cd packages/package-a && npm run test test-subpackage-b: steps: - checkout - run: cd packages/package-b && npm run test test-subpackage-c: steps: - checkout - run: cd packages/package-c && npm run test publish-subpackage-c: steps: - checkout - run: cd packages/package-c && npm run publish workflows: validate-everything: jobs: - validate-everything subpackage-a: jobs: - test-subpackage-a subpackage-b: jobs: - test-subpackage-b subpackage-c: jobs: - test-subpackage-c - publish-subpackage-c
Obviously there are many problems with this, for a start all the CI is defined in one file at the root of the project. Worse, each of the jobs may take some time to complete and on every commit to the project, circleci will run every single job no matter whether there are any changes to the respective subpackages or not. It may not look so bad in this tiny example but the bigger the configuration gets, the worse it is to deal with.
.circle/config.yml is always the same:
version: 2.1 setup: true orbs: circletron: firstname.lastname@example.org workflows: trigger-jobs: jobs: - circletron/trigger-jobs
The trigger job step will take many individual
circle.yml distributed within the project and combine them into a single configuration which will be issued via the continuation API. It will modify the configuration to ensure that jobs that are not necessary are no longer run, in a way that is friendly to CI branch protection rules.
The single configuration file can now be split up across the monorepo, with one optional
circle.yml in the root of the project and where the CI for each subpackage lives in the directory for that subpackage.
In this instance the
circle.yml in the root of the project will host configuration not specific to any subpackage:
version: 2.1 jobs: validate-everything: steps: - checkout - run: npm run general-validation workflows: validate-everything: jobs: - validate-everything
The jobs here are run on every commit and it's also a good place to set
version. It's also a good place to provide
commands that can be used in subpackage specific
Now lets look at
jobs: test-subpackage-a: steps: - checkout - run: cd packages/package-a && npm run test workflows: subpackage-a: jobs: - test-subpackage-a
Everything related to this package all in one place. Even better, when the PR contains no changes within
packages/package-a the jobs for this subpackage will be skipped. You will probably want to use branch protection rules to ensure that when the
test-subpackage-a job fails the PR will be blocked so omitting this job entirely would not be ideal. Omitted jobs remain in
pending state permanently, blocking the PR. circletron replaces unneeded jobs with a simple job using the
busybox:stable docker image that echoes
"Job not required" and issues a success error code.
What if the code in
packages/subpackage-c uses code from
packages/subpackage-a? In this case the jobs within this package should run when the code in
subpackage-a changes, even if the code in the subpackage itself doesn't change.
There may also be some instances where a job should run on every push.
The configuration at
packages/subpackage-c/circle.yml shows how to add dependencies and create jobs which run unconditionally:
dependencies: - subpackage-a jobs: test-subpackage-c: steps: - checkout - run: cd packages/package-b && npm run test publish-subpackage-c: conditional: false steps: - checkout - run: cd packages/package-c && npm run publish workflows: subpackage-c: jobs: - test-subpackage-c - publish-subpackage-a
In this instance
test-subpackage-c will only be run when changes to
package-c are detected and
publish-subpackage-c will run on every push.
CircleCI does not pass the target branch to workflows or jobs so it becomes necessary to help circletron out. By default circletron considers the branches
develop and any branch starting with
release/ as a target branch. The latest commit from the branch commit history which belongs to one of these branches is considered to be the branch-point. For pushes to one of the branches above, all of the jobs are run. This can be changed via the configuration file at
Any branch which matches this regex is considered a target branch, the above configuration shows the default.
circletron is available to use now.