Manage complex development projects by triggering pipelines from other pipelines

It is no secret that software development is becoming an increasingly complex process. The individual elements of software like apps, libraries, and services are interconnected and dependent on many other elements. Development teams deal with a whole ecosystem of services that they develop, maintain, or depend on, which in turn are dependent on other software ecosystems, maintained by separate teams. Maintaining this ecosystem is as complex as you might imagine. Making sure the whole system works, and keeps working as intended, is a hugely challenging effort.

Triggering pipelines from other pipelines is one of many techniques that can help you manage complex connected and interdependent development projects. In this tutorial, I will cover some use cases for using this techniques and why it can be helpful. You can follow along as I present an example of how to implement a pipeline-to-pipeline trigger using CURL.

Prerequisites

This intermediate to advanced tutorial requires some understanding of CircleCI and its pipelines. It will be especially useful if you use CircleCI and have some projects that have already been configured. Everything covered in this article will work on all CircleCI plans.

Use cases for pipeline-to-pipeline triggers

There are several reasons to consider triggering new pipelines from existing pipelines. Microservices orchestration is a prime example. It is a common pattern to have microservices in each respective version control repository, each of which requires a separate CircleCI pipeline configuration. As one service gets deployed, you could redeploy others that depend on this new version. Or you could trigger an expansive suite of integration tests that run independently of the wider test and deployment flow.

If you are not using microservices, you can use pipeline-to-pipeline triggers to publish and test libraries that are integrated with downstream consumers.

Implementing pipeline triggers

One workflow for triggering pipelines follows these steps:

• The first pipeline is executed, makes a build, runs some tests, and triggers a deployment.
• After that process completes successfully, the first pipeline makes an API call to CircleCI to trigger another pipeline in a different project.
• The second pipeline executes a build, runs tests, and triggers a deployment.

In this scenario, the first pipeline can also pass some pipeline parameters like the name and version of what was deployed, so that the newly triggered pipeline has some context.

Coordinating separate services using pipeline triggers

To orchestrate two separate services, you can invoke the CircleCI API from a single job. You will need to know the name of the project with the pipeline you want to trigger from that job. To address a pipeline with an API, you will need the organization name (zmarkan in my case) and the project name (pinging-me-softly because I have an odd sense of humor).

Next, get a personal API token. You must have push access to both projects for this to work. You can store that API token in this project’s environment variable. For increased security, store it in a context. In my example, I used a context called circleci-api, with a CIRCLE_API_TOKEN name for that environment variable.

To trigger a pipeline, you will need to send a POST request to the pipeline’s endpoint of the project. This is the URL I used for this tutorial:

https://circleci.com/api/v2/project/gh/zmarkan/pinging-me-softly/pipeline

The payload must contain details of the branch or tag you want to trigger the pipeline with, along with any pipeline parameters you want to pass to it.

Here is an example job:

  trigger-new-pipeline:
      docker: 
        - image: cimg/base:2021.11
      resource_class: small
      steps:
        - run:
            name: Ping another pipeline
            command: |
              curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan/pinging-me-softly/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"image-name":"'$DOCKER_LOGIN"/"$CIRCLE_PROJECT_REPONAME":"0.1.<< pipeline.number >>'"}}'

The job uses the cimg/base image and the small resource class, which is more than sufficient for sending a single HTTP request. The only dependency is on the CURL tool which ships with all CircleCI images, including cimg/base.

To trigger the pipeline, there is a single command to run: curl. The API key is passed in the Circle-Token header, passing in the environment variable that contains the token. The payload body is a JSON object, which contains a branch and parameters:

  { 
    "branch":"main",
    "parameters": {
        "image-name":"'$DOCKER_LOGIN"/"$CIRCLE_PROJECT_REPONAME":"0.1.<< pipeline/ number>>'"
    }
  }

The value of the image-name parameter is a string constructed of three variables:

• $DOCKER_LOGIN is an environment variable set in this project or in context.
• $CIRCLE_PROJECT_REPONAME is a built-in environment variable that specifies this project in CircleCI.
• pipeline.number is another built-in environment variable.

As a whole, this value provides the name of the Docker image we built in the previous step. I used zmarkan/demo-full:0.1.119 in this example.

Handling the pipeline trigger

The pipeline trigger is handled for you automatically by CircleCI, the same way it handles a new git commit in your repository. By default, all your workflows will run, and it will take the last commit of a branch or tag that you specified in the API call. For this tutorial, I specified main.

You can handle any parameters you pass (image-name in my case) in your jobs and workflows. Just make sure to declare them in the parameters section of the config:

  version: 2.1

  parameters:
    image-name:
      type: string
      default: "Not a real image name"

  jobs:
    print-image-name:
      docker: 
        - image: cimg/base:2021.11
      steps:
        - checkout
        - run:
            name: Echo Image name
            command: echo << pipeline.parameters.image-name >>

  workflows:
    run-workflow:
      jobs:
        - print-image-name

This pipeline example has a single workflow. It prints out the name of the image we built in the previous pipeline and passed via the API to this pipeline.

Conclusion

In this article I have demonstrated how to use the CircleCI API to trigger another pipeline when the first project’s pipeline is complete. It is a straightforward process that requires a single API call from a running job, and is available to all CircleCI users and organizations.

This technique opens up new possibilities for orchestrating complex workflows, including microservices architecture, and decoupling expensive integration testing work from build and deploy scenarios. There are many, many more possibilities for you and your team to experiment with.

If you have any questions or suggestions about this article, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.