Automate Container Image Patching with Copacetic and GitHub Actions

In an era where software deployment speed and security are paramount, automating critical tasks in the development pipeline is essential. GitHub Actions offers a robust platform to achieve this automation seamlessly.

In this article, we'll walk you through the creation of a GitHub Actions workflow that focuses on automating the patching and signing of container images using a CNCF sandbox project Copacetic.

Use copa-action to automate patching

Copacetic is a command-line application that uses reports from vulnerability scanners, such as trivy, to directly patch containers images.

It's able to do this by parsing the vulnerability reports generated by the scanners and creating additional layers on top of the container image that includes patches to the CVEs identified by the scanner.

For my Microsoft Build talk, I had to setup a shell task in the workflow to run the copa CLI, but luckily for you an official action now exists. Here's how you set it up.

# .github/workflows/patch.yml

on:
    push:
      branches:
          - copaGitOps

jobs:
      patch:
          runs-on: ubuntu-latest

          strategy:
            fail-fast: false
            matrix:
                  # provide relevant list of images to scan on each run
                  images: ['s3cexampleacr.azurecr.io/azure-voting-app-rust:v0.1.0-alpha']

          steps:
          - name: Login to Azure
            id: login
            uses: azure/login@v1
            with:
              creds: ${{ secrets.AZURE_CREDENTIALS }}

          - name: Login to Azure Container Registry
            run: |
                az acr login --name ${{ vars.ACR_NAME }}

          - name: Set up Docker Buildx
            uses: docker/setup-buildx-action@v3.0.0

          - name: Generate Trivy Report
            uses: aquasecurity/trivy-action@69cbbc0cbbf6a2b0bab8dcf0e9f2d7ead08e87e4
            with:
              scan-type: 'image'
              format: 'json'
              output: 'report.json'
              ignore-unfixed: true
              vuln-type: 'os'
              image-ref: ${{ matrix.images }}

          - name: Check Vuln Count
            id: vuln_count
            run: |
              report_file="report.json"
              vuln_count=$(jq '.Results | length' "$report_file")
              echo "vuln_count=$vuln_count" >> $GITHUB_OUTPUT

          - name: Extract Patched Tag 
            id: extract_tag
            run: |
              imageName=$(echo ${{ matrix.images }} | cut -d ':' -f1)
              current_tag=$(echo ${{ matrix.images }} | cut -d ':' -f2)

              if [[ $current_tag == *-[0-9] ]]; then
                  numeric_tag=$(echo "$current_tag" | awk -F'-' '{print $NF}')
                  non_numeric_tag=$(echo "$current_tag" | sed "s#-$numeric_tag##g")
                  incremented_tag=$((numeric_tag+1))
                  new_tag="$non_numeric_tag-$incremented_tag"
              else
                  new_tag="$current_tag-1"
              fi

              echo "patched_tag=$new_tag" >> $GITHUB_OUTPUT
              echo "imageName=$imageName" >> $GITHUB_OUTPUT

          - name: Copa Action
            if: steps.vuln_count.outputs.vuln_count != '0'
            id: copa
            uses: project-copacetic/copa-action@v1.0.0
            with:
              image: ${{ matrix.images }}
              image-report: 'report.json'
              patched-tag: ${{ steps.extract_tag.outputs.patched_tag }}
              buildkit-version: 'v0.11.6'
              # optional, default is latest
              copa-version: '0.3.0'

          - name: Docker Push Patched Image
            id: push
            if: steps.login.conclusion == 'success'
            run: |
                # docker push ${{ steps.copa.outputs.patched-image }}
                echo "DIGEST=$(docker push ${{ steps.copa.outputs.patched-image }} | grep -oE 'sha256:[a-f0-9]{64}')" >> $GITHUB_OUTPUT

There's a lot going on here, so let's walk through the steps in the job.

The workflow uses a matrix strategy with a list of images to generate a new job per image in the array of images.

A private Azure Container Registry is used the workflow authenticates to azure and then runs the az acr login command to populate the docker login credentials. Which allows you to use docker commands to interact with the registry.

The Trivy action is used to generate a list of vulnerabilities that exist for the targeted container image and outputs that to a report.json file to be used for patching by Copacetic.

A few lines of Bash are used to populate a variable that contains the number of vulnerabilities found.

Extract Patch Tag is another shell task that uses for awksedfu to determine what the container image patched tag should be following the format tag-1, with -1 indicating the image has been patched.

Copa action installs and runs the Copa CLI to ingest the reports.json and patch the remote container image, which results in a local version of the image that's been patched.

1. Once copa is done and a newly patched image has been created locally, the docker push command is used to push the image to the container registry, but it also grabs the digest from the image.

Curious what the digest is used for? Keep reading to find out. :)

Signed patched container images with Notation

Notation is another CNCF project that just hit version v1.0.0 in Aug. 202 and it allows you digitally sign artifacts. Those signatures become stamps of approval for those digital artifacts letting you know you can trust them to run in your environment.

And if you're using admission controllers on your cluster with projects like Ratify, those signatures are even more important because without them your containers won't be allowed to deploy on the cluster.

At any rate, adding the notary-actions to your GitHub Action Workflow is fairly straight forward:

- name: Setup Notation
  if: steps.push.conclusion == 'success'
  uses: notaryproject/notation-action/setup@v1
  with:
    version: "1.0.0"

- name: Notation Sign
  if: steps.push.conclusion == 'success'
  uses: notaryproject/notation-action/sign@v1
  with:
    plugin_name: azure-kv
    plugin_url: https://github.com/Azure/notation-azure-kv/releases/download/v1.0.1/notation-azure-kv_1.0.1_linux_amd64.tar.gz
    plugin_checksum: f8a75d9234db90069d9eb5660e5374820edf36d710bd063f4ef81e7063d3810b
    key_id: ${{ vars.KEY_ID }}
    target_artifact_reference: ${{ steps.extract_tag.outputs.imageName }}@${{ steps.push.outputs.DIGEST }}
    signature_format: cose
    plugin_config: |-
        name=${{ vars.CERT_NAME }}
        self_signed=false

notation-action/setup installs the notation CLI on the GitHub action runner at the specified version.

notation-action/sign' optionally installs a plugin, in this case, the azure-kv plugin. It then takes the necessary arguments as parameters and then passes them to thenotation sign` command that digitally signs the target artifact (container image)

Test the Workflow

If you've followed along with each tutorial in this series, you have the Flux resources for the Azure Voting app deployed along with Flux resources for the image automation to detect when new container images are pushed to the container registry. And now, you have a workflow that patches the Azure Voting app container images.

So, now the fun begins. Once you commit and push the .github.com/workflows/patch.ymlfile to the repository it will patch the Azure voting container image and push it to the container registry. Once that image is pushed the Flux image automation will detect that a new version of the image exists. When that happens the image automation will update the kustomization.yaml with the newer tag, which will in turn trigger the Flux resources monitor your application's repo to redeploy the application to the Azure Kubernetes Cluster.

Run the following commands & watch the magic happen:


git add . && git commit -m 'Add copa patching workflow' && git push