Reducing docker images size

You might have been building containers for a long time, and in all your previous builds, the security and/or the size of your containers wasn't the priority. But if you got here now you are giving priority to those topics.

For now we can focus first on the container size, and for this lets talk about a couple of options here.

• Build upon a minimal base image.
• Use multi-stage builds.
• Create containers with statically linked applications.

Build upon a minimal base image

Instead of using a big image like ubuntu, debian, etc. Which could be a huge image for what is needed, we can use some alternatives that will allow us to start from a much smaller base.

debian latest d91720f514f7 124MB
ubuntu latest 216c552ea5ba 77.8MB

alpine latest 9c6f07244728 5.54MB
gcr.io/distroless/static-debian11 latest 561bdfb51245 2.35MB

The alpine and google distroless container are way smaller that ubuntu or debian.

In the case of google distroless container the only problem will be the package management for that container, because it wont have any package manager, another thing “missing” will be a shell. But they have a good number of base images for several runtime.

gcr.io/distroless/static-debian11
gcr.io/distroless/base-debian11
gcr.io/distroless/cc-debian11
gcr.io/distroless/python3-debian11
gcr.io/distroless/java-base-debian11
gcr.io/distroless/java11-debian11
gcr.io/distroless/java17-debian11
gcr.io/distroless/nodejs-debian11

Alpine on the other hand is a distro container, therefore it has a package manager and a shell, but it is small enough to be use has base container.

Lets start with an easy go application, you can see the source code of the application here. The container will expose a web application by default in the port 5000 with a general information of the system.

If we start with the basic way of building this container, will be something like this.

FROM golang

WORKDIR /app

COPY . /app/

RUN go mod download; \
    CGO_ENABLED=0 go build -ldflags="-s -w" -o bce -v .

EXPOSE 5000

ENTRYPOINT ["/app/bce"]

And after this we can build the container with the command:

docker build -t highercomve/bce .

After this we will end with a whooping 1.02GB size for the container.

highercomve/bce latest 614d597dbfb5 4 seconds ago 1.02GB

That doesn’t sound that good. Lets now change the the base container from golang to golang:alpine.

FROM golang:alpine

WORKDIR /app

COPY . /app/

RUN go mod download; \
    CGO_ENABLED=0 go build -ldflags="-s -w" -o bce -v .

EXPOSE 5000

ENTRYPOINT ["/app/bce"]

And with only this change we now are in only 375MB

highercomve/bce latest 1fb5114c906e About a minute ago 375MB

Same container, same application a lot less space.

Use multi-stage builds

Now the second optimization we can do is use a multi-stage build. For this we need to mark in the FROM command the name the stage will have, an then we use another base as the final runtime of the container.

FROM golang:alpine as builder # FIRST STAGE

WORKDIR /app

COPY . /app/

RUN go mod download; \
    CGO_ENABLED=0 go build -ldflags="-s -w" -o bce -v .

FROM gcr.io/distroless/static-debian11

WORKDIR /app

COPY --from=builder /app/bce /app/bce # IMPORT FROM THAT STAGE
COPY static /app/static/
COPY templates /app/templates/

EXPOSE 5000

ENTRYPOINT ["/app/bce"]

Now after this change we can see the difference it will be huge, because the resulting image will only be 8.73MB in size.

highercomve/bce latest e68c3ebb73e8 About a minute ago 8.73MB

Create containers with statically linked applications

Another thing you could do is build the application with the libraries linked and use the scratch base container.

FROM golang:alpine as builder

WORKDIR /app

COPY . /app/

RUN go mod download; \
    CGO_ENABLED=0 go build -ldflags="-s -w -extldflags=-static" -o bce -v .

FROM scratch

WORKDIR /app
COPY --from=builder /app/bce /app/bce
COPY static /app/static/
COPY templates /app/templates/

EXPOSE 5000
ENTRYPOINT ["/app/bce"]

With the change we will reduce another 2MB in size

highercomve/bce latest f32c277bf5a6 7 seconds ago 6.39MB

Now as a bonus i will add how to use this techniques to build a multi arch build container. Maybe you have the same application but you need to run it in arm or riscv architecture. For this we can use the buildx plugin from docker https://github.com/docker/buildx.

And for the same container i will use two arguments that are going to be injected by buildx.

ARG TARGETPLATFORM
ARG BUILDPLATFORM

And will allow us to take decisions about how to configure the environment variables for golang. I will use this build.sh script to map the target platform variable injected by buildx to the golang environment variables for building.

#!/bin/sh
set -e

echo "Building for $TARGETPLATFORM" 
export CGO_ENABLED=0

case "$TARGETPLATFORM" in
 "linux/arm/v6"*)
  export GOOS=linux GOARCH=arm GOARM=6
  ;;
 "linux/arm/v7"*)
  export GOOS=linux GOARCH=arm GOARM=7
  ;;
 "linux/arm64"*)
  export GOOS=linux GOARCH=arm64 GOARM=7
  ;;
 "linux/386"*)
  export GOOS=linux GOARCH=386
  ;;
 "linux/amd64"*)
  export GOOS=linux GOARCH=amd64
  ;;
 "linux/mips"*)
  export GOOS=linux GOARCH=mips
  ;;
 "linux/mipsle"*)
  export GOOS=linux GOARCH=mipsle
  ;;
 "linux/mips64"*)
  export GOOS=linux GOARCH=mips64
  ;;
 "linux/mips64le"*)
  export GOOS=linux GOARCH=mips64le
  ;;
 "linux/riscv64"*)
  export GOOS=linux GOARCH=riscv64
  ;;
 *)
  echo "Unknown machine type: $machine"
  echo "Building using host architecture"
esac

go mod download
go build -ldflags="-s -w -extldflags=-static" -o bce -v .

We need to be sure of one thing, the compilation process needs to run in the same architecture as the host building the container, if not we may lose performance trough emulation. For this we can use the BUILDPLATFORM argument in the FROM definition on the dockerfile.

Something like this.

FROM --platform=$BUILDPLATFORM golang:alpine as builder

ARG TARGETPLATFORM
ARG BUILDPLATFORM

WORKDIR /app

COPY . /app/

RUN /app/build.sh

FROM scratch

WORKDIR /app

COPY --from=builder /app/bce /app/bce
COPY static /app/static/
COPY templates /app/templates/

EXPOSE 5000

ENTRYPOINT ["/app/bce"]

With this new Dockerfile we can build the container using the buildx plugin.

docker buildx build --platform linux/arm64 -t highercomve/bce --load .

In here we introduce 3 new arguments for the build command:

• platform: in here we can defined a list separated by coma of several architectures
• load: this argument configure docker buildx to load the image to our local docker environment after the build process is done.
• push: this will push to docker hub all the container images after the building process is done.

You can build several platform in one command and push to docker hub.

docker buildx build --platform linux/arm64,linux/amd64,linux/arm --push -t highercomve/bce .

Docker hub will support one images with multiple architectures and when some device pull with one of those architecture will get the correct one without any problem.

If we want to test the container running with one architecture different for the one of our system we will need to load the qemu binaries.

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

And after that run the container with the platform we like

docker run -it -p 5000:5000 --platform linux/arm64 highercomve/bce

In this example the output of the web page should be something like this

I hope this could be useful for you and help you to maintain more optimized containers

May the force be with you