Splitting .circleci/config.yml

In this post I introduce how to split a huge .circleci/config.yml.

CircleCI doesn't support to split .circleci/config.yml, so we manage all workflows and jobs configuration into one file .circleci/config.yml.
If the repository is Monorepo, the more the number of services increases, the more the size of .circleci/config.yml becomes large and it's hard to maintain .circleci/config.yml.
By splitting .circleci/config.yml per service, it makes easy to maintain .circleci/config.yml and we can configure split file's CODEOWNERS.

To split .circleci/config.yml, you have to generate .circleci/config.yml by merging split files and commit both split files and .circleci/config.yml.

circleci config pack

We can merge split files with the command circleci config pack, but I introduce the other tool circleci-config-merge.

CircleCI CLI is an official tool so it's reliable, but I feel the restriction of the file name and the directory structure is a little strict.
We have to manage all files on the same directory, and the file path is reflected to generated YAML content.

For the detail of circleci config pack, please see the official document.

https://circleci.com/docs/2.0/local-cli/#packing-a-config

If you can accept the restriction of circleci config pack, I recommend to use it because it is an official tool.
But if it is difficult to accept the restriction, maybe circleci-config-merge would help you.

circleci-config-merge

circleci-config-merge is a CLI tool to generate .circleci/config.yml by merging split files.

The usage of circleci-config-merge is like the following.

$ circleci-config-merge merge <file1> [<file2> ...]

There is no restriction of file paths, and the format of split file is same as .circleci/config.yml.

For example, you can manage files on the same directory.

.circleci/
  config.yml # generated
  src/
    service1.yml # split config per service
    service2.yml
    ...

Or you can also manage files on each service directory.

service1/
  circleci/
    workflow.yml # you can split file freely
    jobs.yml
  ...
service2/
  circleci/
    config.yml
  ...
...

circleci-config-merge merges the list of workflow jobs.

For example,

workflows:
  build:
    jobs:
    - foo

workflows:
  build:
    jobs:
    - bar

The workflow build's jobs are merged as the following.

workflows:
  build:
    jobs: # sort by job name for comparison
    - bar
    - foo

Test .circleci/config.yml in CI

If you split .circleci/config.yml, you should test in CI whether .circleci/config.yml is generated by merging split files.
circleci-config-merge doesn't provide such a feature, but you can implement the test with the other tool like dyff.

I have created an example repository suzuki-shunsuke/example-circleci-config-merge.
You can use this example as a reference to split .circleci/config.yml and setup CI.

Use case

Lastly, I introduce a use case of circleci-config-merge.
Recently, I split a huge .circleci/config.yml which is over 6,000 lines to about 60 files.
It was hard to maintain the original .circleci/config.yml, but by splitting it became easy to maintain .circleci/config.yml.
If you are suffer from a huge .circleci/config.yml, let's split it!

Conclusion

In this post I introduced how to split a huge .circleci/config.yml.
We can generate .circleci/config.yml by merging split files with circleci-config-merge.
Please see the example suzuki-shunsuke/example-circleci-config-merge as a reference to split .circleci/config.yml and setup CI.