With modern applications, Docker is a common tool on the tool belt.
We use Docker to develop and test our application and to run it in production.
Have you ever thought about the security of Docker containers in production?
While Container security is a wide field, let's focus on building more secure images for production deployment without spending hours online researching best practices.
This blog post introduces a low-friction approach to finding potential vulnerabilities in your images.
Let's say we're working on an API service written in Rust. The repository contains a
Dockerfile with just a handful of instructions. It might look like this:
FROM rust:1.70 EXPOSE 8000 COPY ./ ./ RUN cargo build --release CMD ["./target/release/app"]
At first glance, the file looks straightforward. Docker Hub offers an official image for almost every programming language. The official Rust image has a full toolchain preinstalled. We can use this image right away to compile and run the project—a perfect fit for a development environment.
As soon as we go into production, however, things look different.
We don't need the compiler toolchain when working with a compiled language. The Rust binary is self-contained, only requiring a libc runtime (which often is already present).
No matter which language we work with, a production image must satisfy different requirements than a development image, such as installing specific package versions, strict permissions, etc.
Another aspect is image size. A development image can easily take up lots of disk space. Consider this example where an image based on
rust:1.70 takes up 3GB:
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE rust-dev latest 3ad31037e881 26 seconds ago 3.07GB
The compiler toolchain eats up some of the space, but the build artifact folder is most likely responsible for most of it. We need neither to run the application in production.
Let's optimize our
Dockerfile. To reduce image size, we implement a multi-stage build. With this approach, we still use the official Rust image to build the application. Once compiled, we copy the binary out of it into a small production image that only has a few things installed and configured. These changes lead to a much smaller disk space footprint.
FROM rust:1.70.0 as builder WORKDIR /usr/app RUN USER=root cargo new --bin pokemon_api WORKDIR /usr/app/pokemon_api COPY ./Cargo.toml ./Cargo.toml RUN cargo build --release RUN rm src/*.rs COPY ./src ./src # 5. Build for release. RUN rm ./target/release/deps/pokemon_api* RUN cargo build --release #-------- FROM debian:bookworm-slim EXPOSE 8000 RUN apt-get update RUN apt-get install -y curl && rm -rf /var/lib/apt/lists/* WORKDIR /usr/app COPY --from=builder /usr/app/pokemon_api/target/release/pokemon_api /usr/app/pokemon_api ADD ./rocket_config.toml Rocket.toml CMD ["/usr/app/pokemon_api"]
With our new production image in place, let's find out if there's more we can optimize.
In the following steps, we use a local Kubernetes cluster (such as kind) to test the image.
With the cluster up and running, let's install some tooling to help us with image scanning.
In this case, we're using KubeClarity. Follow the installation instructions in the README to install it into your development cluster.
With KubeClarity installed, let's deploy our service.
(Find the example source code here.)
kubectl create ns pokemon kubectl apply -n pokemon -f k8s/deployment.yaml
As mentioned, this service is written in Rust, performing a simple HTTP Call to fetch a Pokemon CSV, which returns JSON data.
Upon further inspection of the source code, you'll notice it uses
curl to perform the HTTP request.
With this, we're simulating the code's dependency on specific system packages (
Before we deploy this service in other environments, such as staging or production, we want to find out if there are any potential security problems with this particular image.
If you haven't already, in a separate terminal window, run the following command:
kubectl port-forward -n kubeclarity svc/kubeclarity-kubeclarity 9999:8080
Then, head over to
On the top, select
pokemon as namespace and click on Options to open the settings dialogue.
In the settings dialogue, select CIS Docker Benchmark.
Save your changes and click Start Scan.
After a few seconds, the scan will conclude and show us a summary of the findings.
http://localhost:9999/applicationResources, click on ghcr.io/schultyy/rust-pokemon-api:0.0.1 in the list.
On the detail page, select CIS Docker Benchmark.
The list shows seven findings with varying severities.
This step reduces the image size and cleans up superfluous and unneeded caches.
Check out this list for more details.
ADD both have an overlap in features. Prefer to use
COPY only supports basic file-copying mechanisms.
ADD, however, has additional features that are not immediately obvious, such as
tar extraction or remote URL support.
See docs.docker.com for more details.
apt-get update with
apt-get install, Docker will cache the update layer and reduce the total number of layers.
Consider this example:
FROM ubuntu:22.04 RUN apt-get update RUN apt-get install package-a
In this case, the first time you run
docker build, it executes every command and caches every line. The next time you run
docker build, Docker determines nothing has changed and will finish much faster.
You return to this
Dockerfile a few days later, realizing you need to install an additional package.
FROM ubuntu:22.04 RUN apt-get update RUN apt-get install package-a package-b
build, Docker realizes the first line hasn't changed and will use a cached layer. Distributions like Debian and Ubuntu update repositories frequently (i.e. deleting old package versions).
Package sources updated by
apt-get update from two days ago might point to a package or version that no longer exists, causing
apt-get install to fail.
If you combine both lines, however, you will get updated package sources every time the list of packages to install changes.
Running your application as
root comes with several security implications. Therefore, dropping privileges as soon as possible is a good practice.
RUN groupadd -g 10001 appuser && \ useradd -u 10000 -g appuser appuser \ && chown -R appuser:appuser /app USER appuser:appuser
All instructions after
USER will run as
appuser, without privileges.
It's a good practice to include a Health Check within the
Dockerfile to allow Docker to determine if the container is healthy.
This addition is helpful when the container is running but the application within the container has crashed.
HEALTHCHECK CMD curl --fail http://localhost:3000 || exit 1
See docs.docker.com for more information.
With all discussed findings implemented, our Dockerfile now looks like this:
FROM rust:1.70.0 as builder WORKDIR /usr/app RUN USER=root cargo new --bin pokemon_api WORKDIR /usr/app/pokemon_api COPY ./Cargo.toml ./Cargo.toml RUN cargo build --release RUN rm src/*.rs COPY ./src ./src # 5. Build for release. RUN rm ./target/release/deps/pokemon_api* RUN cargo build --release #-------- FROM debian:bookworm-slim EXPOSE 8000 HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 CMD [ "curl --fail http://localhost:8000/health" ] RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* WORKDIR /usr/app COPY --from=builder /usr/app/pokemon_api/target/release/pokemon_api /usr/app/pokemon_api COPY ./rocket_config.toml Rocket.toml RUN groupadd -g 10001 appuser && \ useradd -u 10000 -g appuser appuser \ && chown -R appuser:appuser /usr/app USER appuser:appuser CMD ["/usr/app/pokemon_api"]
Next, we build a new image version:
docker build -t ghcr.io/schultyy/rust-pokemon-api:0.0.2 .
Once built, let's push it to the registry:
docker push ghcr.io/schultyy/rust-pokemon-api:0.0.2
With the new image version published to the container registry, let's deploy it into the test cluster:
kubectl apply -f k8s/deployment.yaml
As a last step, we re-run the scan (with the steps above) to verify we have no more fatal findings:
Performing these kinds of checks is only of KubeClarity's features. Whenever you run a scan, it also scans for vulnerable packages. If you want to learn more about package scanning, check out this blog post.
Also, make sure to check out the KubeClarity GitHub project!