Getting started with scheduled pipelines

CircleCI’s scheduled pipelines let you run pipelines at regular intervals; hourly, daily, or weekly. If you have used scheduled workflows, you will find that replacing them with scheduled pipelines gives you much more power, control, and flexibility. In this tutorial, I will guide you through how scheduled pipelines work, describe some of their cool use cases, and show you how to get started setting up scheduled pipelines for your team. I will demonstrate using both the API and the UI and how you can set up scheduled pipelines from scratch or migrate to them from scheduled workflows.

Prerequisites

Here is what you will need to follow the tutorial:

• CircleCI account with a project configured
• Some Node.JS knowledge

You can use the sample project as you go through the steps.

Scheduling regular builds and tests

There are many reasons to schedule CI/CD work instead of just executing when you push code to a repository. The first and most obvious reason to schedule is to have regular builds of software. Often folks outside of the development team (product managers, QA engineers, and other stakeholders) need access to the latest build of software you and your team are working on. Of course, you can always trigger a build manually on demand, but that can be distracting for you and other developers on the team. It is so much easier to automate this process and point all the stakeholders to where they can find the newest build. This applies to everything from web apps and mobile apps, to backend services, libraries, and anything in between.

Scheduled builds automatically build software at night or off-hours when there is no development going on. These nightly builds (as they are sometimes called) take the latest revision off your main branch and produce a staging or beta build of the software.

You can use scheduled pipelines to run the entire test suite so the nightly build also verifies that your software works and is ready and waiting when you start your next working day. Scheduling allows you to run those expensive and time-consuming functional or integration tests you do not want to run on every commit. The build does not need to be run nightly. You can run it at a cadence that suits your team, every few hours, or even several times per hour.

Scheduling other kinds of work

But why stop at builds? You might also schedule work that is not necessarily a build but needs to happen regularly, like a batch process, database backups, or restarting services. If it can be added to a script on a machine that has access to your environments, it can be done in a CI/CD system.

Legacy way of scheduling — scheduled workflows

The ability to schedule work isn’t exactly new to CircleCI. Development teams have been able to include scheduling as part of the configuration, with schedules defined using cron syntax on the workflow level. There are a few downsides to this approach, though:

• It requires development work to make any changes to the schedule, or even to review schedules already in place.
• In CircleCI, pipelines are triggered, not workflows.
• Fine-tuning permissions was difficult because it was not always clear who scheduled the work that was triggered.

How scheduled pipelines improve developer experience

Here are some of the benefits of scheduled pipelines:

• You can schedule entire pipelines, including any pipeline parameters you want to pass.
• With scheduling handled outside the configuration file, you can query and set schedules using both the API and the UI. This ability provides more flexibility with who manages scheduling and execution.
• Scheduled pipelines work with contexts. Contexts give you fine-grained control of who has access to perform and schedule certain jobs. You can gate your deployment credentials to only the engineers with sufficient permissions, so no one else can set up those schedules. Contexts can also be used with dynamic configuration to unlock even more flexibility for your CI/CD setup.

Now that we have covered some basic facts about scheduled pipelines, we can implement one.

Implementing a scheduled pipeline

First, we need a pipeline to schedule. Luckily, I have a project that had used scheduled workflows. It is an open source Android library project that runs nightly deployments, so it is ideal for scheduling use cases. I recommend that you fork the project on GitHub and set it up as a project on CircleCI.

For our example schedule, we want to run a build every night and deploy it to the Sonatype snapshots repository. This build makes the repository available for anyone to use and get the freshest code.

There is a workflow already defined for it in our .circleci/config.yml - nightly-snapshot:

  parameters:
    run-schedule:
        type: boolean
        default: false

  nightly-snapshot:
    when: << pipeline.parameters.run-schedule >>
    jobs:
      - android/run-ui-tests:
          name: build-and-test
          system-image: system-images;android-23;google_apis;x86
          test-command: ./gradlew assemble sample:connectedDebugAndroidTest
      - deploy-to-sonatype:
          name: Deploy Snapshot to Sonatype
          requires:
            - build-and-test

The workflow has two jobs: one to run the tests and another to deploy the snapshot. There is no scheduling logic in the pipeline itself, apart from a when statement that checks for the run-schedule pipeline parameter. Our scheduler will set the parameter when triggering the pipeline.

Using the API to schedule pipelines

To get started with scheduling using the API, you need the API token. To get the token, log in to CircleCI and click your avatar in the bottom left corner. Clicking your avatar opens User Settings. Navigate to Personal API Tokens, create a new token, and save it somewhere safe. In the sample project there is a build-scheduling directory, with a file called .env.sample. You can copy that file to .env and replace the placeholder token with yours. You should do the same with other parts of the .env file: PROJECT_ID and ORG_NAME.

CIRCLECI_TOKEN=YOUR_CIRCLECI_TOKEN
PROJECT_ID=YOUR_PROJECT_ID
ORG_NAME=YOUR_VCS_USERNAME
VCS_TYPE=github

With environment variables set, we can make the API calls. Scheduled pipelines use the CircleCI API v2. To post a new schedule, you need to make a POST request to the endpoint. The endpoint consists of your CircleCI project, your username, and your VCS provider. Here is an example: https://circleci.com/api/v2/project/github/zmarkan/android-espresso-scrollablescroll/schedule

This example POST call from the setup_nightly_build.js script uses Axios JavaScript library:

const token = process.env.CIRCLECI_TOKEN

axios.post("https://circleci.com/api/v2/project/github/zmarkan/android-espresso-scrollablescroll/schedule", {
    name: "Nightly build",
    description: "Builds and pushes a new build to Sonatype snapshots every night. Like clockwork.",
        "attribution-actor": "system",
        parameters: {
          branch: "main",
          "run-schedule": true
        },
        timetable: {
            per_hour: 1,
            hours_of_day: [1],
            days_of_week: ["TUE", "WED", "THU", "FRI", "SAT"]
        }
    },{
        headers: { 'circle-token': token }
    }
)

The body payload contains the schedule name, which must be unique, and an optional description. It includes an attribution actor, which can be either system for a neutral actor or current, which takes your current user’s permissions (as per the token you use). The payload also includes parameters like which branch to use when triggering the pipeline and any other parameters you have set up. This tutorial uses run-schedule in the config file. The last part of the body is the timetable, which is where we define when and how frequently to run our scheduled pipelines. The fields to use here are per_hour, hours_of_day, and days_of_week. Note that this does not take a cron expression, which makes it more easily parsable by humans reasoning with the API.

In the headers, we will pass a circle-token using the token we generated earlier in the CircleCI UI.

In the timetable, we set everything to run at 1:00 am (UTC), on Tuesday to Saturday, the night after the work has finished. We do not need to run on Sunday and Monday, because in our example project, there is no one working over the weekend. The codebase will not change on those days.

In addition to the POST method, the API takes exposes other methods such as GET, DELETE, and PUT for retrieving, deleting, and updating schedules. There are samples for get_schedules.js and delete_schedule.js in the repository.

Using the GUI

Instead of using the API, you can set up scheduled pipelines from right in the CircleCI dashboard. From your project in CircleCI, go to Project Settings, and select Triggers from the menu on the left.

Click Add Scheduled Trigger to open the page where you can set up a new scheduled pipeline. The form has the same options as the API: trigger name, days of the week, start time, parameters, attributed user, and the others.

Click Save Trigger to activate it. The trigger will be ready to start scheduling your pipelines at the dates and times you specified.

Migrating from scheduled workflows

So far, we have explored setting up the pipelines and reviewing them using both the API and the GUI. Now we can focus on migrating your existing scheduled workflows to the more advantageous scheduled pipelines.

This example shows a scheduled workflow where everything is defined in the config file. It includes the trigger configuration to the workflow definition, passing in the schedule with the cron expression:

nightly-snapshot:
    triggers: #use the triggers key to indicate a scheduled build
      - schedule:
          cron: "0 0 * * *" # use cron syntax to set the schedule
          filters:
            branches:
              only:
                - main
    jobs:
      - android/run-ui-tests:
          name: build-and-test
          system-image: system-images;android-23;google_apis;x86
          test-command: ./gradlew assemble sample:connectedDebugAndroidTest
      - deploy-to-sonatype:
          name: Deploy Snapshot to Sonatype
          requires:
            - build-and-test

In our case, this is run at midnight every day, and we want to trigger it only on the main branch.

To migrate there are a few steps to complete:

1. Trigger the existing nightly-snapshot workflow in the scheduled pipeline.
2. Introduce a new pipeline variable called run-schedule, as we did in the first example.
3. For all workflows, add when expressions that indicate to run them when run-schedule is true and not to run other workflows unless run-schedule is false.

parameters:
  run-schedule:
    type: boolean
    default: false

workflows:
  build-test-deploy:
    when:
      not: << pipeline.parameters.run-schedule >>
    jobs:
      ...

  nightly-snapshot:
    when: << pipeline.parameters.run-schedule >>
    jobs:
      ...

The rest of the process is the same as when setting up from scratch:

1. Resolve the cron expression; you can use an online tool like this.
2. Set up the schedule using either the API or the GUI with the new timetable configuration, and the run-schedule pipeline parameter.

Conclusion

Scheduled pipelines are a versatile component of your CI/CD toolkit, allowing you to:

• Trigger your builds on a recurring basis
• Utilize pipeline parameters
• Fine-tune control of who can set up pipelines
• Clearly define the permissions needed when they run using CircleCI contexts

In this tutorial, you have learned about how scheduled pipelines work in CircleCI and how to set them up, either from scratch or porting from scheduled workflows. We also covered how to use the API and the web UI, and reviewed some use cases for them to get you started.

Let us know how you and your team fare with scheduled pipelines, porting your scheduled workflows, or if you would like to see any additions to the UI or API features.

If you have any feedback or suggestions about topics I should cover next, reach out to me on Twitter - @zmarkan.