How to reduce compile time using Docker as dev environment on C++

Abstract

In this post I'll show how to avoid to wasting your time rebuilding a fresh new image every time you make a small change in your code. The reference language is C ++ but the proposed solution can be used for every compiled languages and project which require time consuming building process.

Problem

Best practices suggest to use multi-stage Dockerfile copying all the source code in the builder stage, compile it and then copy the binary file on next stage. This approach makes sense in production instead while you're in dev mode and you're forced to use the container to compile and makes test has a great problem. Every time you make a small change in the source code this means a fresh new Docker image; even if the rebuild uses previous layer and just recompile whole the project this can be a problem if it is time consuming requiring several minutes to recompile.

If you're in dev mode waiting for several minutes every time you make a simple change can be annoying.

Idea

It will be great to recompile only the files with the changes just like when you're developing and compiling on your system. In order to do this the source code is stored inside the host's file system and mounted as volume by the container. This guarantee that all the object files are not looses and b reused on next compile processes.

Hence the idea is to have a container with the following features:

• have all the necessary to compile
• recompile only the changes on source code
• generate the binary file on host's file system
• there is no need to rebuild the image to generate the binary file.

This container will be a "binary generator" and it's purpose its to generate a new binary file without recompile whole the project. Then use new binary file to create the runtime docker image.

Solution

In order to better explain my solution I have created a simple "Hello world" project that you can find here. The tree of the project is the following:

    .
    ├── core.cpp
    ├── core.hpp
    ├── dev_env
    │   ├── builder
    │   │   ├── docker-compose.yml
    │   │   └── Dockerfile
    │   └── runtime
    │       ├── docker-compose.yml
    │       └── Dockerfile
    ├── docker-compose.yml
    ├── Dockerfile
    ├── main.cpp
    ├── Makefile
    └── README.md

The Dockerfile and docker-compose.yml at the root level show a single stage build process that will show the original problem: changing one file will rebuild the the whole process, which is what we are trying to avoid.

In the dev_env directory we can find the proposed solution:

• the builder directory contains the Dockerfile and docker-compose to generate the builder container (the one used to generate the binary file).
• the runtime directory contains all the necessary to build test or runtime image. Instead the runtime contains all the necessary to build test or runtime image.

Builder directory

docker-compose.yml

version: '2.3'

services:
   
    binary_builder:
        container_name: "binary_builder"
        image: binary_builder
        build:
            dockerfile: ./dev_env/builder/Dockerfile
            context: ../../
            args:
                src_path: /src
        volumes:
            - ../.././:/src:rw

As you can see on the src directory, on container, is mounted the root of the project, source code included.

Dockerfile

    FROM ubuntu:18.04
    ARG src_path=/src
            
    RUN apt-get update                          && \ 
        apt-get install -y                         \
        g++ build-essential
     
     RUN mkdir -p $src_path
     WORKDIR $src_path
     CMD     make

The Dockerfile is quite simple it takes the source code from $src_path and the compile it (if necessary) every time it will be started. Another advantage is that it is not necessary to rebuild the builder container to generate a new binary.

Runtime directory

The Dockerfile is very simple, use the same base image used in builder level and copy the binary file from the host file system.

Dockerfile

    FROM ubuntu:18.04 as run_time
    COPY  hello_world /hello_world
    WORKDIR /
    CMD ./hello_world

The advantage is that to rebuild the runtime container you have to generate a new binary file and copy it on the new image saving time.

Conclusion

I hope this article has helped you.