Docker Security: Clair

Docker offers an amazing solution for packaging applications along with their dependencies. You can find the application itself, its supporting elements like Maven or NPM packages, the base Operating System, and other necessary tools like Java and NodeJS in Docker images. By creating a build pipeline, you can very easily create a Docker image whenever there are code changes. The best part is that docker images contain all the dependencies, allowing you to deploy them on any platform that supports Docker, be it an on-prem Kubernetes setup or a cloud-based platform like AWS ECS.

Docker Image Scanning

Since docker images contain all the required OS files for the application to run, it is very essential to carefully examine all the packages installed in the docker image and be sure that there are no vulnerabilities. Docker's philosophy is creating lightweight and minimal containers that serve a specific purpose. In other words, it is recommended to always create separate images for different applications. This approach perfectly aligns with the microservice ecosystem, where each microservice has its own dedicated docker image.

Image Scanning in Registries

The easiest way of scanning docker images is scanning them inside of registries. Both Dockerhub and Quay offer built-in image scanning capabilities, but there are a few limitations to keep in mind. Currently, DockerHub's scanning feature is only available for private repositories by default. On the other hand, Quay's image security scanning is exclusive to Quay Enterprise.

Clair - Open Source Image Scanner

Clair, developed by CoreOS, is a fantastic open source vulnerability scanner specifically designed for docker images. It is capable of gathering vulnerabilities from various vulnerability databases for different operating systems like Debian, Ubuntu, Red Hat, Alpine, and Oracle Linux. The best part is that Clair can be easily obtained as a docker image, enabling one-off scans to be seamlessly integrated into the build pipeline. However, when running Clair for the first time, it needs to download vulnerability information, which can be quite time-consuming, taking around 20-30 minutes. This delay becomes problematic when implementing a continuous integration and continuous deployment (CICD) pipeline.

The arminc/clair-db image on Docker Hub solves this. It runs a daily build of the vulnerability database and creates pre-populated database images that are ready to use right away. By utilizing this pre-built database, the need for time-consuming downloads during the initial setup of Clair is eliminated. This makes it incredibly convenient for seamless integration into CICD pipelines.

To run the pre-built database, simply follow these steps:

docker run -d --name db arminc/clair-db

Clair requires a config.yaml file that contains config like the DB password to get started. Another docker image, still from arminc solves this, providing a default config which is embedded in Clair. The command below can be run to connect with our db.

docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan

How Clair works

Clair exposes an API to scan the individual docker layers. To be able to run a scan, a Clair client is needed which can do the job. A useful tool for this is Klar, which is a popular CLI client written in Go language which can run point and shoot scans. However, it needs the image to be scanned to be inside of a registry already and cannot scan images that are locally stored. To use image scanning in a CICD pipeline, it is always better to run scans locally on an image before pushing it to the registry. Having results available before release allows us to choose whether we can still proceed and push that image or break the build based on the number and severity of the vulnerabilities in the docker image. An alternative, that can still help us run local scans is Clair-Scanner.

We can run clair-runner tool by providing it the IP of the docker bridge gateway to run a local scan, by running this command:

# get docker gateway IP
DOCKER_GATEWAY=$(docker network inspect bridge --format "{{range .IPAM.Config}}{{.Gateway}}{{end}}")
# download scanner cli
wget -qO clair-scanner https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 && chmod +x clair-scanner
# run scan on local image - dubu in this case
./clair-scanner --ip="$DOCKER_GATEWAY" dubu:latest

Running a local scan

To be able to run a local scan on a docker image before pushing it. You can use this docker-compose.yml script

version: '3'

services:
  db:
    image: arminc/clair-db
    container_name: clairdb
    restart: always

  clair:
    image: arminc/clair-local-scan
    container_name: clairlocal
    depends_on:
      - db
    ports:
      - "6060:6060"
    restart: always

By using docker-compose, it streamlines the process and ensures that the linked containers (if we create more than 1) can communicate seamlessly. Now let's look at our bash script that can automate this

#!/usr/bin/env bash

# Function to check if a command executed successfully
check_command() {
    if [ $? -ne 0 ]; then
        echo "Error executing the previous command!"
        exit 1
    fi
}

# Start the Clair and database containers
docker-compose up -d
check_command

# Give some time for the database to initialize. 
# Note: It's better to have a health check to confirm when the db is ready, but for simplicity, we use sleep here.
sleep 20

# Download and prepare the clair-scanner
if [ ! -f clair-scanner ]; then
    wget -qO clair-scanner https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
    check_command
    chmod +x clair-scanner
fi

# Get Docker's bridge network gateway
DOCKER_GATEWAY=$(docker network inspect bridge --format "{{range .IPAM.Config}}{{.Gateway}}{{end}}")
check_command

# Scan the specified Docker image
./clair-scanner --ip="$DOCKER_GATEWAY" dubu:latest
check_command

Let us break down the bash script

1. Function check_command: This function checks the exit status of the last executed command. If the command failed (i.e., didn't return a status of 0), it outputs an error message and exits the script. This provides better error handling throughout your script.

2. Download Check: The script checks if clair-scanner already exists before attempting to download it. This can save time if you're running the script multiple times.

3. Health Check Note: I've added a note in the script to emphasize that using a health check for the database would be a more robust solution than just waiting a set amount of time with sleep.

Please note that after copying this script, to execute it, you need to make sure it has execute permissions (chmod +x your_script_name.sh) and then run it (./your_script_name.sh).

Conclusion: CICD Integration

The whole plan is for us to be able to integrate image scanning with out CICD pipeline. To do this, you can use this script below as part of your Jenkins pipeline.

stage("docker_scan"){
    steps {
        script {
            // Ensure the containers are not already running (cleanup from a previous run).
            sh 'docker rm -f db || true'
            sh 'docker rm -f clair || true'

            // Start the Clair database and wait for it to initialize.
            sh 'docker run -d --name db arminc/clair-db'
            sh 'sleep 15' // Consider a more robust health check here.

            // Start the Clair container.
            sh 'docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan'
            sh 'sleep 1'

            // Download the clair-scanner if not already present.
            sh '''
                if [ ! -f clair-scanner ]; then
                    wget -qO clair-scanner https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
                    chmod +x clair-scanner
                fi
            '''

            // Get Docker's bridge network gateway and run the scanner.
            sh '''
                DOCKER_GATEWAY=$(docker network inspect bridge --format "{{range .IPAM.Config}}{{.Gateway}}{{end}}")
                ./clair-scanner --ip="$DOCKER_GATEWAY" myapp:latest || exit 0
            '''
        }
    }
    post {
        always {
            // Clean up, remove the Clair and database containers.
            sh 'docker rm -f db'
            sh 'docker rm -f clair'
        }
    }
}

clair-scanner by default will break the build if any issues are found. In this code we ignore the exit code by appending || exit 0.

Let us take a look at out Jenkins build