CMake on SMT32 | Episode 7: unit tests

It's been almost four years, and it's about time to revive this series on CMake and STM32 (or MCUs in general, STM32 just serve as a tangible example).

Funny enough, I'm not working with microcontrollers anymore.... This is a very recent change in my professional career, even think I will probably work again with MCUs sooner or later. However, I still use with CMake to build C++ projects! The real reason to resume this series is my desire to write an article Docker for builds. In 2020, I had never used Docker at all. Today, I use it very often, and I'm convinced it could have solved some build and environment issues back then.

But hey, you might be wondering why I'm talking about Docker. Isn't this article supposed to be about unit tests? Yep, it is. I originally planned to cover unit tests back in 2020, but things didn't pan out. It seems logical to write about this topic first. It will give us even more reasons to use Docker (subscribe so you don't miss the next episode)!

Why unit tests?

I don't want to get into a lengthy debate about pros & cons about unit tests. Sure, they can be misused (so bad), but I believe they are a very powerful tool to improve software quality. There are tons of great articles on this topic out there if you want to if you want to delve into this topic. If you speak (or at least understand) French, I was invited on a podcast to talk about unit tests, and I explained why I believe they are a crucial part of software development.

After 13+ years as a software developer, mostly working projects with MCUs (and especially STM32), I must say that unit tests aren't very common in the embedded world. Embedded projects sometimes feel more like electronics-with-some-code projects code than modern software projects. Hardware can be a real constraint too, so we tend to think that unit tests aren't for us.

In this article, I want to show that it doesn't have to be necessarily that way. With CMake, we can easily build unit tests for our embedded projects.

Running tests on board

Since we're dealing with electronic devices, our first thought is naturally to run tests directly on the board.

Running unit tests on board is great, because the same compiler and processor are used for both the tests and the real application. The tests are very close to the real usage of the code.

To do this, we will need to create an additional firmware dedicated solely to running tests and generating a test report. This means we have to add another add_executable() in your CMakeLists.txt. Similar to the real application's target, we have call target_compile_definitions(), target_include_directories(), target_link_options(), and so on, to generate a suitable firmware for the board. This process is not different from what we covered in the previous episodes.

However, running tests on board presents several challenges:

• A test framework that compiles for the target must be found.
• Tests into may need to be split into several executables so that each one fits within the flash memory of the MCU.
• The standard output of your board must be captured to get the test report.
• Running tests takes time, simply because your have to program the board.
• Integration in CI pipeline may be problematic because it may not be feasible to connect a board to the server.

While these challenges are not insurmountable, the following section will explore the advantages of running tests on your computer, eliminating these constraints and offering additional benefits.

If you want to learn more about embedded tests, the great MCU on Eclipse blog has recently published an article on this topic.

Running tests on your computer

Instead of running the tests directly on board, an alternative approach is to compile them using a compiler for PC and execute them on our computer. This eliminates the constraints mentioned earlier.

You may argue that we are using different compiler and means it won't accurately reflect of the real usage of the code. And in some ways, you're absolutely right. Indeed, even something as "basic" as the size of an integer could differ between arm-none-eabi-gcc and regular gcc.

But that's actually the silver lining! Compiling our project with another compiler may raise new warnings, highlighting non-portable constructs. Running our code on a different CPU may reveal undefined behaviors in our code. Finally, being able to compile a significant part of the project for another target demonstrates that the abstraction from the hardware layers are well-designed. Naturally, this also implies that we won't be able to test the parts that are tightly coupled to hardware features.

In the remainder of this article, I will demonstrate how to use CMake to build an embedded firmware and PC-based tests within the same CMakeLists.txt.

Project structure

The structure of the project is as follows:

Most elements are simply the result of the previous episodes. I have just added a directory (tests) and two files (compute.hpp and test_compute.cpp), because... we need something to test!

A simple test case

The purpose here is to demonstrate some CMake features, don't expect amazing code.

First, we need something to test. For instance a simple quadratic function implementation. This is a good candidate for PC-based tests, because it doesn't have any dependencies to the BSP. The code goes into compute.hpp:

namespace compute {

template <typename T>
T quadratic(T a, T b, T c, T x) {
    return a * x * x + b * x + c;
}

}

The tests are written in a separate file, test_compute.cpp. For this example, I have decided a use Catch2, a great unit testing framework for C++ that I have been using for the last 5 years.

#include <catch2/catch_test_macros.hpp>

#include "compute.hpp"

TEST_CASE("Compute quadratic function") {
    int a = 2;
    int b = 1;
    int c = 10;

    CHECK(compute::quadratic(a, b, c, 0) == 10);
    CHECK(compute::quadratic(a, b, c, 1) == 13);
    CHECK(compute::quadratic(a, b, c, 2) == 20);
    CHECK(compute::quadratic(a, b, c, -3) == 25);
}

True story: I found that my function had a bug when I wrote these tests. I had mistakenly written a * a * x instead of a * x * x in the implementation of compute::quadratic().

Create the executable for the tests

Similar to adding tests for on-board execution, we have to add another add_executable() in our CMakeLists.txt. However, things are sightly different: we need to handle two different compilers now. Fortunately, CMake makes it straightforward to determine whether we are cross-compiling (to create our STM32-based firmware) or not (to create our PC-based program), thanks to the variable CMAKE_CROSSCOMPILING. We will see in the next section that is variable is set when we provide CMake with a toolchain file.

The structure of our CMakeLists.txt will look like this:

cmake_minimum_required(VERSION 3.15.3)

project(nucleo-f413zh)

enable_language(C CXX ASM)
set(CMAKE_C_STANDARD 99)
set(CMAKE_C_STANDARD_REQUIRED ON)
set(CMAKE_C_EXTENSIONS OFF)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

if(CMAKE_CROSSCOMPILING)
        message(STATUS "Cross compiling for board...")

        # Here goes the code from the previous episodes

else()
        message(STATUS "Compiling for PC...")

        # Here goes the code to build the unit tests
        # See below :)
endif()

Catch2 is available on GitHub. You can get a copy of its source code and integrate it into your project source tree. You can also use FetchContent module to get it from GitHub directly during the build process, as suggested in Catch2's official documentation. We will use this technique here.

Let's complete the else() clause from the previous snippet to create our executable:

message(STATUS "Compiling for PC...")

# Get Catch2
include(FetchContent)

FetchContent_Declare(
        Catch2
        GIT_REPOSITORY https://github.com/catchorg/Catch2.git
        GIT_TAG v3.5.2
)

FetchContent_MakeAvailable(Catch2)

# Create executable
add_executable(tests
        tests/test_compute.cpp)

target_include_directories(tests PRIVATE
        source)

target_link_libraries(tests PRIVATE
        Catch2::Catch2WithMain)

Build and run the tests

As mentioned before, passing a toolchain file to CMake from the command-line automatically set CMAKE_CROSSCOMPILING. We will hence have to generate two "projects" from a single CMakeLists.txt.

On one side, we generate a project for the arm-none-eabi-gcc toolchain, just like in the first episode of this series. From the nucleo directory:

$ cmake -B build-embedded -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=arm-none-eabi-gcc.cmake
-- The C compiler identification is GNU 10.3.1
-- The CXX compiler identification is GNU 10.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/arm-none-eabi-gcc - skipped
...
-- Cross compiling for board...
...

$ cmake --build build-embedded/
Scanning dependencies of target nucleo-f413zh.out
[  3%] Building C object CMakeFiles/nucleo-f413zh.out.dir/BSP/Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_tim.c.obj
[  7%] Building C object CMakeFiles/nucleo-f413zh.out.dir/BSP/Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_tim_ex.c.obj
[ 11%] Building C object CMakeFiles/nucleo-f413zh.out.dir/BSP/Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_uart.c.obj
...
[100%] Linking CXX executable nucleo-f413zh.out
   text    data     bss     dec     hex filename
   7868      20    2892   10780    2a1c nucleo-f413zh.out
[100%] Built target nucleo-f413zh.out

On the other side, we generate another project to build with our default system compiler:

$ cmake -B build-tests -DCMAKE_BUILD_TYPE=Release                                             
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
...
-- Compiling for PC...
...

$ cmake --build build-tests/
[  0%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2.dir/catch2/benchmark/catch_chronometer.cpp.o
[  1%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2.dir/catch2/benchmark/detail/catch_analyse.cpp.o
[  2%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2.dir/catch2/benchmark/detail/catch_benchmark_function.cpp.o
[  3%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2.dir/catch2/benchmark/detail/catch_run_for_at_least.cpp.o
...
 96%] Built target Catch2
[ 97%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2WithMain.dir/catch2/internal/catch_main.cpp.o
[ 98%] Linking CXX static library libCatch2Main.a
[ 98%] Built target Catch2WithMain
[ 99%] Building CXX object CMakeFiles/tests.dir/tests/test_compute.cpp.o
[100%] Linking CXX executable tests
[100%] Built target tests

Because our executable depends on Catch2, CMake builds the library before.

Our tests are now ready to run:

$ ls build-tests/
CMakeCache.txt  CMakeFiles  Makefile  _deps  cmake_install.cmake  tests

$ ./build-tests/tests 
Randomness seeded to: 179865564
===============================================================================
All tests passed (4 assertions in 1 test case)

Hurray! The tests passed!

Conclusion

In this episode, we have discussed unit tests for embedded projects.

We have seen that running tests on board is great but there are some difficulties to overcome. Running tests on PC don't have these difficulties and may even offer additional benefits. If you have the chance to do both, do it!

We've seen how to manage both an STM32-based firmware and a PC-based program within the same CMakeLists.txt. We generate two separate build directories, one for each compiler and hence one for each executable.

I hope that this episode convinced you to add unit tests to your STM32 (or any other MCU) projects. Happy tests!