Advanced pipeline orchestration with the circleback pattern

This tutorial covers:

1. Orchestrating triggers for a dependent pipeline
2. Intro to the circleback pattern
3. Creating the config for a circleback pipeline

With multiple teams working on many projects, having a single pipeline for your software is just not enough. These projects need to be built and integrated before they can be tested and released. So how do dev teams handle this situation? Many teams approach the problem by breaking down software into smaller parts that do less, and are easier to maintain and build. This approach has resulted in the microservices architectures that are increasingly common in our industry.

The downside is that breaking software into smaller parts increases complexity. The combination of more parts and higher complexity makes it more difficult to test applications and operate them in production.

CI/CD pipelines are not an exception here. In my previous article in this series, we looked at triggering pipelines from other pipelines. That post described how to link multiple projects together in CircleCI, so that one deployment kicks off another.

In this article we will go a few steps further. I will provide an example of pipeline orchestration where the first pipeline not only triggers the other, but also waits on another pipeline to complete, before continuing. I call this the circleback pattern because it “circles back” to the second pipeline to signal when it has completed the work. Using this pattern allows for better integration testing and more complex deployment and release scenarios.

Prerequisites

• This tutorial covers advanced pipeline orchestration techniques, so experience with CircleCI and DevOps is a must.
• The tutorial also builds on a previous tutorial: Triggering pipelines from other pipelines.

Resources and sample code

The source code for this example is located in two repositories:

1. Pipeline A - does the triggering

2. Pipeline B - gets triggered and circles back.

Introducing the circleback pattern

There are three steps in the circleback orchestration of these multiple pipelines:

1. The first pipeline waits
2. The second pipeline terminates
3. Results are retrieved from the first pipeline Another way of describing it is that the second pipeline calls or “circles back” to the first, while the first one is suspended and waiting for the signal to continue.

An alternative approach would be to keep the first pipeline running, checking periodically whether the second pipeline has finished. The main drawback of this approach is the increased cost for the running job’s continuous polling.

Depending on the status of the second pipeline, the first continues executing until it either terminates successfully or fails.

Why have pipelines wait for other projects to complete?

As mentioned in the introduction, this is an advanced technique aimed at taming the complexity that stems from working with multiple projects. It can be used by teams working on multiple interdependent projects that they do not want to put in a single repository like a monorepo. Another use would be to have a centralized repository for tests, for example in a hardware company. This technique could also be useful for integration testing of a microservices application, or for orchestrating complex deployment scenarios. There are many possibilities.

Implementing pipeline triggers

We have 2 pipelines we want to orchestrate

1. Pipeline A, which does the triggering
2. Pipeline B is triggered, and circles back to pipeline A

Pipeline B is dependent on A, and can be used to validate A.

Both pipelines need to have API keys set up and available. You can use the API key set as an environment variable (CIRCLECI_API_KEY) in the job in pipeline A, and also in pipeline B when it calls back. You can either set it in both projects, or at the organization level as a context. For this tutorial, I set it at the organization level as the circleci_api context, so that both projects can use the same API key.

Trigger the pipeline

The triggering process is explained in depth in the first part of this tutorial, Triggering pipelines from other pipelines. In this follow-up tutorial, I will cover just the important differences.

• To circle back from the pipeline, pass the original pipeline’s ID to it. Then it can be retrieved and reached with the API.
• You also need to store the triggered pipeline’s ID. You will need to get its result later on.

In the sample code, the parameter is called triggering-pipeline-id:

curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/circleback-cicd-project-b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}'

To store the pipeline ID, wrap your curl call in $() and assign it to the variable CREATED_PIPELINE. To extract the ID from the response body, use the jq tool, and write it to the file pipeline.txt:

CREATED_PIPELINE=$(curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/semaphore_demo_project_b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
              | jq -r '.id'
              )
              echo "my created pipeline"
              echo $CREATED_PIPELINE
              mkdir ~/workspace
              echo $CREATED_PIPELINE > pipeline.txt

Now that you have the file pipeline.txt created, use persist_to_workspace to store it and use it in a subsequent job:

- persist_to_workspace:
            root: .
            paths: 
              - pipeline.txt  

The whole job configuration is here:

...
jobs:
  trigger-project-b-pipeline:
      docker: 
        - image: cimg/base:2021.11
      resource_class: small
      steps:
        - run:
            name: Ping another pipeline
            command: |
              CREATED_PIPELINE=$(curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/semaphore_demo_project_b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
              | jq -r '.id'
              )
              echo "my created pipeline"
              echo $CREATED_PIPELINE
              mkdir ~/workspace
              echo $CREATED_PIPELINE > pipeline.txt
        - persist_to_workspace:
            root: .
            paths: 
              - pipeline.txt  
...

Orchestrating the waits

The previous job will trigger a pipeline B, which needs to complete before it can circle back to pipeline A. You can use the approval job in CircleCI like this:

...
workflows:
  node-test-and-deploy:
    jobs:
      ...
      - trigger-project-b-pipeline:
          context: 
            - circleci-api
          requires:
            - build-and-test
          filters:
            branches:
              only: main
      - wait-for-triggered-pipeline:
          type: approval
          requires: 
            - trigger-project-b-pipeline
      - check-status-of-triggered-pipeline:
          requires:
            - wait-for-triggered-pipeline
          context:
            - circleci-api
      ...

After the job trigger-project-b-pipeline, enter the wait-for-triggered-pipeline. Because that job type is approval it will wait until someone (in this case, the API) manually approves it. (More details in the next section.) After it is approved, add a requires stanza so it continues to a subsequent job.

Both jobs that use the CircleCI API have the context specified, so the API token is available to both as an environment variable.

Circle back to pipeline A

For now we are done with pipeline A, and it is pipeline B’s time to shine. CircleCI’s approval job is a special kind of job that waits until accepted. It is commonly used to hold a pipeline in a pending state until it is approved by a human delivery lead or infosec engineer.

At this point, pipeline B knows the ID of pipeline A, so you can use the approval API to get it. You only have the ID of the pipeline being run, but not the actual job that needs approval, so you will need more than one API call:

1. Get all jobs in the pipeline
2. Find the approval job by name
3. Send a request to approve the job

Approving the job allows pipeline A to continue.

If the tests fail in pipeline B, then that job automatically fails. A workflow with required jobs will not continue in this case. You can get around by using post-steps in the pipeline, which always executes. The whole workflow is shown in the next sample code block.

Parameters:

parameters:
  triggering-pipeline-id:
    type: string
    default: ""

...

workflows:
  node-test-and-deploy:
    jobs:
      - build-and-test:
          post-steps:
            - approve-job-in-triggering-pipeline
          context: 
            - circleci-api       

Script to perform the approval API call can be implemented like this. For this tutorial, I used a command.

...
commands:
  approve-job-in-triggering-pipeline:
    steps:
      - run:
          name: Ping CircleCI API and approve the pending job
          command: |
            echo << pipeline.parameters.triggering-pipeline-id >>
            if ! [-z "<< pipeline.parameters.triggering-pipeline-id >>"] 
            then
              workflow_id=$(curl --request GET \
                --url https://circleci.com/api/v2/pipeline/<< pipeline.parameters.triggering-pipeline-id >>/workflow \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].id')

              echo $workflow_id

              waiting_job_id=$(curl --request GET \
                --url https://circleci.com/api/v2/workflow/$workflow_id/job \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[] | select(.name == "wait-for-triggered-pipeline").id')

              echo $waiting_job_id

              curl --request POST \
                --url https://circleci.com/api/v2/workflow/$workflow_id/approve/$waiting_job_id \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json"

            fi
          when: always       
...

The script first checks for the existence of the triggering-pipeline-id pipeline parameter. It proceeds only if that parameter exists. The when: always line in the command makes sure that this executes regardless of termination status.

Then it makes 3 API calls:

1. Get workflow ID in a pipeline. There is just one workflow in that pipeline for this sample project.
2. Get jobs in that workflow, use jq to select the one that matches the approval job name (wait-for-triggered-pipeline), and extract the approval job’s ID.
3. Make the request to the approval endpoint with the waiting job ID.

For this tutorial, we are storing results like workflow ID and job ID in local bash variables, and using them in subsequent calls to the API.

Note : If you have more jobs than can be sent in a single response, you might have to handle pagination as well.

Now that you have made the approval request, pipeline B is complete, and pipeline A should be running again.

Update pipeline A with the result of B

After pipeline A has been approved, the next job in the workflow will begin. If your workflow graph requires it, pipeline A can trigger multiple jobs.

We still do not have any indication of the result of the previous workflow. To get that information, you can use the API again to get B’s status from pipeline A. An example job could look like that: check-status-of-triggered-pipeline.

First, you need to retrieve the ID of the triggered pipeline, which is pipeline B. This is the same ID that was persisted in a workspace in an earlier step. Retrieve it using cat:

 - attach_workspace:
          at: workspace
      - run:
          name: Check triggered workflow status
          command: |
            triggered_pipeline_id=$(cat workspace/pipeline.txt)

Then use the API to retrieve the workflow. Use jq to get just the status of the first item in the returned array of workflows:

created_workflow_status=$(curl --request GET \
                --url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].status'
            )

Check that status is not success. If it is not, use exit to terminate the job with exit code -1. If the workflow is successful, it will terminate:

if [["$created_workflow_status" != "success"]]; then
              echo "Workflow not successful - ${created_workflow_status}"
              (exit -1) 
            fi

            echo "Created workflow successful"

Here is the full config for the job check-status-of-triggered-pipeline:

 check-status-of-triggered-pipeline:
    docker: 
      - image: cimg/base:2021.11
    resource_class: small 
    steps:
      - attach_workspace:
          at: workspace
      - run:
          name: Check triggered workflow status
          command: |
            triggered_pipeline_id=$(cat workspace/pipeline.txt)
            created_workflow_status=$(curl --request GET \
                --url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].status'
            )
            echo $created_workflow_status
            if [["$created_workflow_status" != "success"]]; then
              echo "Workflow not successful - ${created_workflow_status}"
              (exit -1) 
            fi

            echo "Created workflow successful"

Conclusion

In this article we have reviewed an example of a complex pipeline orchestration pattern that I have named “circleback”. The circleback pattern creates a dependent pipeline and allows you to wait for it to terminate before completing. It involves making several API keys from both projects, the use of an approval job, and the workspace feature of CircleCI to store and pass values such as pipeline ID across jobs in a workflow. The sample projects are located in separate repositories: project A, and project B.

If this article has helped you in a way, I would love to know, also if you have any questions or suggestions about it, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.