Improve Productivity with CMake and Compiler Cache Integration

Introduction

Build times can significantly impact developers' productivity. While what constitutes a slow or fast build is subjective, it is clear that slow builds introduce longer feedback loops and lead to more context switches, resulting in decreased productivity.

C++ compilers are slow, and the use of the standard C++ library and other popular libraries, especially header-only ones, exacerbates this. Here, you can explore the cost in terms of build time for including certain headers.

A study shows that even modest improvements in build time can positively affect developers' productivity.

Proven Way of Optimizing Build Times

One proven way to optimize build times is by using compiler cache programs such as ccache or sccache. These programs cache compilation results and reuse them, omitting subsequent compiler calls if the inputs have not changed.

According to performance measurements conducted by the ccache team, build time improvements on subsequent compilations can range from 5x to 145x.

From my personal experience, realistic improvements can range from 1.5x to 10x, depending on multiple factors such as the build machine hardware, OS, compiler, and project source code.

So, What's the Deal?

If compiler cache programs can bring such improvements, why aren't they used everywhere for every C++ project? One possible answer is the difficulty of integration. There are multiple build systems and compiler types, each with its own peculiarities when integrating compiler cache programs. The cross-platform nature of modern software adds even more complexity to the integration process.

Nowadays, for many cross-platform C/C++ software projects, CMake is the go-to build system. According to the Stack Overflow Developer Survey 2023, 14.34% of developers use CMake, with only Make being more widely used as a cross-platform C++ build system. However, if you look at the sccache README, there's a lengthy section on how to start using a compiler cache and make it somewhat usable with CMake. This section does not cover all the possible combinations of OSes, compilers, and generators supported by CMake.

Solution for CMake

To solve this problem, the cmake-findccache module was created. As the name suggests, it finds a suitable compiler cache program and configures the CMake project to enable the use of the compiler cache.

It supports Xcode, Ninja, Unix Makefiles, and Visual Studio generators, and has been tested with gcc, clang, and MSVC compilers on Linux, macOS, and Windows.

As shown in the example project, in most cases, only a couple of lines are needed to enable the use of ccache:

list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/..")

find_package(ccache)

By default, the module will only look for ccache. To try other compiler cache programs, define the CCACHE_PROGRAMS variable with the list of desired compiler cache programs in order of preference, for example:

set(CCACHE_PROGRAMS "ccache;sccache;buildcache" CACHE STRING _ FORCE)

If a ccache program is found and the generator is supported, the CMake configuration phase log should contain lines similar to the following:

-- Found ccache: /usr/bin/ccache
-- Using compiler cache: YES

After the first successful build of the project, inspect the ccache statistics by running ccache --show-stats. The result should be similar to the output below. Cacheable calls statistics should be non-zero, and after the first build, most calls will result in cache misses due to an empty cache.

Cacheable calls:      2 /   2 (100.0%)
  Hits:               0 /   2 ( 0.00%)
    Direct:           0
    Preprocessed:     0
  Misses:             2 /   2 (100.0%)
Local storage:
  Cache size (GiB): 0.0 / 5.0 ( 0.00%)
  Hits:               0 /   2 ( 0.00%)
  Misses:             2 /   2 (100.0%)

On subsequent builds, cacheable call statistics should continue to increment, along with the number of hits. For example:

Cacheable calls:      4 /   4 (100.0%)
  Hits:               2 /   4 (50.00%)
    Direct:           2 /   2 (100.0%)
    Preprocessed:     0 /   2 ( 0.00%)
  Misses:             2 /   4 (50.00%)
Local storage:
  Cache size (GiB): 0.0 / 5.0 ( 0.00%)
  Hits:               2 /   4 (50.00%)
  Misses:             2 /   4 (50.00%)

This is when compilation time should decrease, thanks to using cached results instead of invoking the compiler!

But... Still Experimental and Fragile

After examining more combinations of OSes, generators, and compilers, it turned out that in some combinations, the compiler cache does not work as expected:

• On Windows with MSYS2, neither mingw64 nor clang64 works with the Ninja generator, resulting in a ccache: error: Could not find compiler "D:\a\_temp\msys64\clang64\bin\clang++.exe" in PATH error during compilation.
• On Windows GitHub Runner (Windows Server 2019) with the CL compiler and Visual Studio 17 2022 generator, ccache behaves oddly. It caches compilation results from the first run of the build, but for the second run, compilations are not accounted for in the statistics at all. However, on Windows 10 Pro, a similar setup works just fine.

Further Improvements

• Fix or find workarounds for the above issues.
• Add support for more compilers, such as clang-cl.
• Test with other cache tools, like sccache.

Feel free to clone and submit pull requests to the cmake-findccache project to improve compiler cache support in CMake!

Conclusion

Compiler cache support can bring significant build time improvements and increase productivity. However, even with the improved integration between CMake and compiler cache programs provided by cmake-findccache, it remains an experimental or unavailable feature on some platforms and configurations.