Running Jobs in a Container via GitHub Actions Securely

Like any modern CI/CD platform, GitHub allows users to run CI jobs in a container. This is great for running consistent and reproducible CI jobs as well as reducing the amount of setup steps that is required for the job to run (e.g., running actions/setup-python to install Python environment and installing necessary packages via pip) as those environments and dependencies can be baked into the container.

In order to make use of this feature, in the GitHub yaml file, specify the container to run any steps in a job via jobs.<job_id>.container. This will tell GitHub to spin up a container and run any steps in that job to run inside. If you have both scripts and container actions, GitHub will run the container actions as sibling containers on the same network with the same volume mounts.

jobs:
  container-test-job:
    runs-on: ubuntu-latest
    container:
      image: node:18
      env:
        NODE_ENV: development
    steps:
      - name: Check for dockerenv file
        run: (ls /.dockerenv && echo Found dockerenv) || (echo No dockerenv)

While using public images are great, for most non-open-source use cases, you'll need to pull from private registries. To do so, you can pass in a map of username and password like the following:

container:
  image: my-registry/my-image
  credentials:
     username: ${{ github.actor }}
     password: ${{ secrets.github_token }}

Easy, right? But let's take a look at when the above approach can become problematic.

Problem

GitHub's current approach works great if you already have a static password that you can pass in securely via GitHub's secret mechanism. However, if you are dealing with temporarily credentials, then there's no way to pass them in securely currently.

To illustrate, let's take AWS ECR as an example. To grab a private image from ECR, you might have two steps like login-to-amazon-ecr and run-tests.

In the first step, you can use the aws-actions to configure credentials and login to ECR. Finally, you will have to set the username and password in the output to send to the next job run-tests:

jobs:
  login-to-amazon-ecr:
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/my-github-actions-role
          aws-region: us-east-1
      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2
        with:
          mask-password: 'false'
    outputs:
      registry: ${{ steps.login-ecr.outputs.registry }}
      docker_username: ${{ steps.login-ecr.outputs.docker_username_123456789012_dkr_ecr_us_east_1_amazonaws_com }} 
      docker_password: ${{ steps.login-ecr.outputs.docker_password_123456789012_dkr_ecr_us_east_1_amazonaws_com }}

Note the flag mask-password: 'false'. This is because in order for GitHub to make use of this output, it needs to be unmasked. As of the time of writing, masked outputs cannot be passed to separate jobs (see this issue).

This means that while this will technically work, it is insecure as now the docker_password output will be logged unmasked if debug logging is enabled.

  run-tests:
    name: Run tests
    needs: login-to-amazon-ecr
    container:
      image: ${{ needs.login-to-amazon-ecr.outputs.registry }}/my-ecr-repo:latest
      credentials:
        username: ${{ needs.login-to-amazon-ecr.outputs.docker_username }}
        password: ${{ needs.login-to-amazon-ecr.outputs.docker_password }}
    steps:
      - name: Run steps in container
        run: echo "run steps in container"

Solutions

Until GitHub supports a way to either pass masked values as outputs or support a different way to authenticate and pull private images, we have a few options.

Disable debug logging

Currently, anyone who has access to run a workflow can enable step debug logging for a workflow re-run. You could either opt to remove human access to trigger workflows or disable re-runs. While this technically solves the issue, it will severely impact developer productivity and experience in a negative way.

Limit private repos runners can access

We could instead elect to accept the risk of having temporary docker credentials printed out to debug logs for a short duration. As a compromise, you can limit which ECR repositories that GitHub actions can pull from. The rationale here is that if your private container does not contain any confidential IP (e.g., mostly just running tests and setup scripts) then temporarily giving attackers to download or list containers images may be acceptable. To take this to the extreme, you could also consider just using a public repo as well.

Run custom runner images

If neither of those quick-fix solutions are acceptable, then you will need to create customer runner images and bake in the docker login step yourself.

Continuing with our AWS example, we could use the amazon-ecr-credential-helper to automatically set credentials. Just download the binary to the runner image and mount a ~/.docker/config.json file with the following contents:

{
    "credsStore": "ecr-login"
}

Then, you can specify that image to run for repos needing to pull private container images, then once it hits the login step, the above binary will take care of it behind the scenes.

The only potential downside of this approach is that now the login step is abstracted away from developers and other docker-login GitHub actions might conflict.