Introducing the CircleCI Config SDK

We are excited to announce the new CircleCI Config SDK is now available as an open-source TypeScript library. Developers can now write and manage their CircleCI config.yml files using TypeScript and JavaScript.

• NPM Package
• GitHub Repository
• Docs

For developers used to the ecosystem and flexibility of a full-fledged programming language, sometimes YAML can feel limiting or intimidating. With the Config SDK you can define and generate your YAML config from type-safe and annotated JavaScript. You can even take advantage of package management to modularize any portion of your config code for reuse.

When paired with CircleCI’s dynamic configuration, the Config SDK makes it possible to dynamically build your CI configuration at run-time, allowing you to choose what you want to execute based on any number of factors, such as the status of your Git repo, an external API, or just the day of the week.

Getting started

For our example, let’s imagine we manage several Node.js projects that are all built using the same framework and generally require the same CI configuration. So we decided that we want to build a config “template” that all of our projects will use, and which can be centrally managed and updated. We’ll create and publish an NPM package that will generate the perfect config file for all of our Node projects.

Let’s build the config template package, and then build the pipelines that will use it.

Setup

We’re going to start by creating a standard NPM package. You can use TypeScript or JavaScript, but we’ll use JavaScript in this example for the sake of speed. The example shown here is based onthis page from the repo’s wiki.

Begin by initializing a JavaScript project in a new directory.

mkdir <your-package-name>

cd <your-package-name>

npm init -y

npm i --save @circleci/circleci-config-sdk

The @circleci/circleci-config-sdk package will allow us to define a CircleCI config file with JavaScript. While we could simply define a config and export it, we can also take advantage of dynamic config and export a function instead. In our example, we’ll keep it simple and create a config generation function that will take a tag parameter for our deployments, and a path parameter to choose where the config file will be exported to.

Create the app

Create an index.js file and import the CircleCI Config SDK package and Node’s fs package so we can write the config to a file.

const CircleCI = require("@circleci/circleci-config-sdk");
const fs = require('fs');

Next we’ll start building up the components of our config file using the Config SDK. You’ll notice because we are working with a TypeScript-based library, we are able to receive code hints, type definitions, documentation and auto-completion.

Create executor

Given we are building a config for our Node.js projects, we’ll begin by defining the Docker executor our jobs will use. You can pass in the Docker image, resource class, and any other parameters you may want to configure.

// Node executor
const dockerNode = new CircleCI.executors.DockerExecutor(
  "cimg/node:lts"
);

Create jobs

We are building up to a workflow that will test our application on every commit, and deploy it when we provide a certain tag. Like our executor, we’ll define these two jobs, both using the executor we just defined and each with a unique set of steps for their respective purposes.

// Test Job
const testJob = new CircleCI.Job("test", dockerNode);
testJob.addStep(new CircleCI.commands.Checkout());
testJob.addStep(new CircleCI.commands.Run({ command: "npm install && npm run test" }));

//Deploy Job
const deployJob = new CircleCI.Job("deploy", dockerNode);
deployJob.addStep(new CircleCI.commands.Checkout());
deployJob.addStep(new CircleCI.commands.Run({ command: "npm run deploy" }));

Jobs can be instantiated with steps or dynamically added to an existing job like shown above. In this overly simplified example, we lack a caching step, but you can see how we build up elements of our configuration.

Create a workflow

With our jobs defined, it’s time to implement them in a workflow and define how they should run. We mentioned earlier that we want the test job to run on all commits, and the deploy job to only run on the given tag.

Now that we are working with top-level components of our config file, let’s finally define a new CircleCI config object, and name the workflow that we will add our jobs to.

//Instantiate Config and Workflow
const nodeConfig = new CircleCI.Config();
const nodeWorkflow = new CircleCI.Workflow("node-test-deploy");
nodeConfig.addWorkflow(nodeWorkflow);

We are not adding any parameters to our testing job because we want to run it on all commits, so we can directly add it to our config object.

nodeWorkflow.addJob(testJob);

For the deploy job, we need to first define the workflow job so that we can add filters to it. We are going to add one filter now which tells CircleCI to ignore this job for all branches, so it doesn’t execute on every commit. We’ll deal with enabling it for a tag in a moment.

const wfDeployJob = new CircleCI.workflow.WorkflowJob(deployJob, {requires: ["test"], filters: {branches: {ignore: ".*"}}});
nodeWorkflow.jobs.push(wfDeployJob);

Export the config generator function

Now that we have everything defined we are ready to create and export the final piece. We are going to create a function which takes in the tag and path parameters we mentioned earlier and will write what we have defined to a new file.

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
 // next step
}

In the newly created writeNodeConfig function, we’ll add the tag filter being passed in here to the deploy job in our workflow and finally write the config to the file supplied by the path parameter using the generate function on the config object.

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
  wfDeployJob.parameters.filters.tags = {only: deployTag}
  fs.writeFile(configPath, nodeConfig.generate(), (err) => {
    if (err) {
      console.error(err);
      return
    }
  })
}

Here is the full source code, which you can also find in the wiki:

const CircleCI = require("@circleci/circleci-config-sdk");
const fs = require('fs');

// Node executor
const dockerNode = new CircleCI.executors.DockerExecutor(
  "cimg/node:lts"
);

// Test Job
const testJob = new CircleCI.Job("test", dockerNode);
testJob.addStep(new CircleCI.commands.Checkout());
testJob.addStep(new CircleCI.commands.Run({ command: "npm install && npm run test" }));

//Deploy Job
const deployJob = new CircleCI.Job("deploy", dockerNode);
deployJob.addStep(new CircleCI.commands.Checkout());
deployJob.addStep(new CircleCI.commands.Run({ command: "npm run deploy" }));

//Instantiate Config and Workflow
const nodeConfig = new CircleCI.Config();
const nodeWorkflow = new CircleCI.Workflow("node-test-deploy");
nodeConfig.addWorkflow(nodeWorkflow);

//Add Jobs. Add filters to deploy job
nodeWorkflow.addJob(testJob);
const wfDeployJob = new CircleCI.workflow.WorkflowJob(deployJob, {requires: ["test"], filters: {branches: {ignore: ".*"}}});
nodeWorkflow.jobs.push(wfDeployJob);

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
  wfDeployJob.parameters.filters.tags = {only: deployTag};
  fs.writeFile(configPath, nodeConfig.generate(), (err) => {
    if (err) {
      console.error(err);
      return
    }
  });
}

Publish the package

With your index.js file completed with the writeNodeConfig function exported, it’s time to publish the package to your package repository of choice, such as NPM or GitHub.

When complete, you should be able to import your package in other projects, just like we imported @circleci/circleci-config-sdk earlier.

Create a CI pipeline

You now have a published NPM package that can generate a CircleCI config file. We can use CircleCI’s dynamic configuration to pull in this package at run-time and dynamically run our generated config file. We can replicate this basic template across our many similar NodeJS projects, and when we want to update or change our config, we will be able to simply update the package we created.

Create config.yml

As usual, our CircleCI project will require a .circleci directory and a config.yml file inside. The config file in this case will be a basic template used in all of our projects which simply tells CircleCI to enable the dynamic configuration feature for the current pipeline, generate the new config file, and run it. We will also create a dynamic directory that we will use later.

└── .circleci/
    ├── dynamic/
    └── config.yml` \

Use the following example configuration file:

version: 2.1
orbs:
  continuation: circleci/continuation@0.3.1
  node: circleci/node@5.0.2
setup: true
jobs:
  generate-config:
    executor: node/default
    steps:
      - checkout
      - node/install-packages:
          app-dir: .circleci/dynamic
      - run:
          name: Generate config
          command: node .circleci/dynamic/index.js
      - continuation/continue:
          configuration_path: ./dynamicConfig.yml
workflows:
  dynamic-workflow:
    jobs:
      - generate-config

You can find this config file and other examples in the Wiki on GitHub.

This config is a boilerplate responsible for executing our package which contains the “real” logic we want to execute.

Notice the setup key is set to true, enabling dynamic config for the pipeline. Using the Node orb, we install a Node app located in .circleci/dynamic (we’ll come back to this), and run the .circleci/dynamic/index.js in Node. This is what will use the package we wrote earlier and create a new config file at ./dynamicConfig.yml, which will finally be executed by the continuation orb.

We will use a config like this one in all of our Node projects, and it is unlikely to need to be updated or changed often. Rather than modifying this config, we update the package we created earlier.

Create the config app

The last thing to do is build our “Config application”. This is the app responsible for implementing our package and acts as the source of our “real” config we plan on executing. Because in this example we have outsourced the majority of the logic of our config to an external package, in this example, our config app is mostly boilerplate as well.

Change directory into .circleci/dynamic where we will set up our application. Initialize a new repository and install the package you published previously.

npm init -y

npm i <your-package-name>

After running these two commands, you should have a package.json file with a dependency showing your package.

"dependencies": {
  "my-circleci-node-config": "^1.0.0",
}

You can modify the semantic version string to dictate what version of your package should be pulled at run time. This means, you could update your my-circleci-node-config package and, if you choose, have all of your CircleCI projects that utilize this package immediately pick up these changes the next time your CI pipeline is triggered.

Not Recommended: Say you always wanted to pull the latest version of the custom dependency you have created, you could use:

"dependencies": {
  "my-circleci-node-config": "x",
},

To be safer, pull in only minor and patch updates, not major releases.

"dependencies": {
  "my-circleci-node-config": "1.x",
},

Finally, utilize your custom package in the .circleci/dynamic.index.js.

Our package exports a function named writeNodeConfig which takes in the value of the tag we want for triggering deployments, and the path we want to export the config to. We know the path from earlier in our config.yml, we set to be ./dynamicConfig.yml, because we are in the dynamic directory, we will prepend ... For the tag, we’ll use a generic regex string v.*

import writeNodeConfig from '<your/package>';

writeNodeConfig("v.*", "../dynamicConfig.yml")

That is our entire application. We simply need to pull whichever version of our package we desire and invoke it to generate our config file from the template we created in the package.

Running the pipeline

To recap, we have a boilerplate config.yml file which instructs CircleCI to enable dynamic configuration, and uses Node.js to build a new config file at run-time. The Node app responsible for building our new config uses a package dependency which contains the “template” of our desired config. We can now use this package in many different projects and update it centrally. Our projects can either pull in the latest version of this package by specifying an x in the package.json, or we can use tools like dependabot to open a pull request automatically to all of our projects that use this package when it is updated.

With dynamic configuration and the Config SDK together, the possibilities are endless. The tutorial above was based on this page from the wiki on GitHub. Check out the rest of our wiki and documentation for even more examples of interesting things you can do with the Config SDK.

We’d love to hear from you and see how you utilize the Config SDK in your own pipelines. Connect with us on Twitter, say hello in our discussion forum, or chat with us on Discord.