Programming in C++

Unit Testing with Catch2

Gerald Senarclens de Grancy

Purpose of Unit Tests

What is a Unit Test

  • Allows automated testing of independent code units
  • No replacement for other test categories
  • Write them as early as possible - if possible before the code it tests
  • Tests can serve as useful requirements specification
  • Create a test for each bug you encounter (refacturing!)

Which problems does unit testing solve?

Code without tests (aka. legacy code) is extremely difficult to work with

Unit tests...

  • are a first line of defense against bugs
  • improve code quality by dictating testable code
  • allow releasing higher quality early and often
  • give more confidence to programmers
  • are a prerequisit to refactoring
  • can be used as a contract with regard to desired behavior
  • prevent regressions (bugs in features priorly working properly)

Which problems does unit testing cause?

Essentially none, but writing unit tests ...

  • seems to cost time
  • demands discipline
  • is far from trivial
  • does not substitute other kinds of testing

What is a Test(case)

  • Answers a single question about the code it is testing
  • Run completely by itself (automation)
  • Results must not require human interpretation
  • Should be independent of other tests and their order of execution
  • Can serve as documentation of how (not) to use your code

Flaky Tests

  • Have different outcomes without changes to the code
  • Erode confidence in testing processes
  • Fix them (mute / skip them until they are fixed) or delete them

Software Quality

User's view (visible to the user)

  • Functionality
  • Reliability
  • Usability
  • Performance/ efficiency

Producer's view

  • Portability
  • Maintainability
  • Testability (vs. legacy code)
  • Transparency (is the code readable/ understandable?)

We Need a Unit Testing Framework

For writing, compiling and running unit tests, a unit testing framework is required.

There are many concurring unit testing frameworks for C++.

At the time of this writing, Boost Test, Catch2 and GoogleTest stand out in terms of quality and features.

Example: Refactor a Function

#include <stdint.h>
// Return the sum of all values from 1 to n.
uint64_t add_to(uint32_t n) {
    uint64_t sum = 0;
    for (uint32_t i = 1; i <= n; ++i) {
        sum += i;
    }
    return sum;
}
Download before_refactoring.cpp

After experiencing performance problems, we want to use a faster algorithm

#include <stdint.h>
// Return the sum of all values from 1 to n.
uint64_t add_to(uint32_t n) {
    return (uint64_t) (n / 2.0) * (n + 1);
}
Download after_refactoring.cpp

How can we apply the change without risking to destroy our application?

  1. Define good test cases
  2. Implement test cases using your project's unit testing library
  3. Run the tests before the change to ensure they pass
  4. Apply the change, then run the tests again

1. Define Good Test Cases

Using pencil and paper / brainstorming: define relevant test cases

  • All code paths should be used
  • Add one or more common cases
  • Include relevant edge cases

25 is a possible common case. Add 26 as an even input. The edge cases should include 0 and 4294967295.
Expected results for these are 325, 351, 0 and 9223372034707292160.

Let's also add 500000000 (expected result 125000000250000000) as 4294967295 does not terminate with the current implementation.

2. Implement defined test cases

#include <cstdint>
#include "catch_amalgamated.hpp"
uint64_t add_to(uint32_t n);  // usually in header file
TEST_CASE("sum_to must calculate correctly", "[sum_to]") {
  // setup code that runs before each section is written here
  SECTION("regular use cases") {
    REQUIRE(add_to(25) == 325);  // if REQUIRE fails, TEST fails immediately
    CHECK(add_to(26) == 351);  // if CHECK fails, TEST will fail, but continue
    REQUIRE(add_to(500000000) == 125000000250000000);
  }
  SECTION("corner cases") { REQUIRE(add_to(0) == 0); }
}
TEST_CASE("sum_to must terminate for all valid input data", "[sum_to_slow]") {
  // does not terminate for the old code version due to an overflow of the
  // loop counter (the loop would terminate only at UINT_MAX + 1)
  REQUIRE(add_to(UINT32_MAX) == 9223372034707292160);
}
Download test_sum_to.cpp

3. Run the Unittests Before the Change

Download catch_amalgamated.cpp and catch_amalgamated.hpp
clang++ --std=c++17 before_refactoring.cpp \
  test_sum_to.cpp catch_amalgamated.cpp -o run_tests
./run_tests # does not terminate; interrupt via [Ctrl]+[C]

What to do?

./run_tests --help
./run_tests --list-tests  # list all available tests
./run_tests [sum_to]  # or ./run_tests "sum_to must calculate correctly"
./run_tests [sum_to] --section "corner cases"

4. Apply the Change, then Run the Tests Again

For the sake of the presentation, instead of applying the change, we use another source file.

clang++ --std=c++17 after_refactoring.cpp \
  test_sum_to.cpp catch_amalgamated.cpp -o run_tests
./run_tests
Download after_refactoring_fixed.cpp

Now the tests are fast, but they are failing. Fix the code, then compile and run the tests again.

Catch2

Amalgamated Version in Version 3

The "old" way of using Catch2 requires no installation at all

Include a single header, compile a single C++ file and link that file

clang++ -c --std=c++17 catch_amalgamated.cpp  # compile Catch2 once
clang++ -c --std=c++17 tested_source_file.cpp
clang++ -c --std=c++17 test_file.cpp
# link amalgamated version of Catch2
clang++ -o your_test catch_amalgamated.o tested_source_file.o test_file.o

This obviously works with any build system.

Recommended Usage

Since version 3 of Catch2, the library consists of many different files

Only inlcude what you need and use cmake to compile your tests

// #include "your_header.hpp"
#include <catch2/catch_test_macros.hpp>

TEST_CASE("trivial 'testcase' with v3", "[tc-tag]") {
  REQUIRE(1 == 1);  // use assertions here and in SECTIONs
}
Download test_v3.cpp

Recommended Usage

cmake automatically downloads Catch2 to your project when needed

cmake_minimum_required(VERSION 3.18)
cmake_policy(SET CMP0048 NEW)
project(sample_test VERSION 1.0)
set(CMAKE_CXX_STANDARD 20)
include(FetchContent)
FetchContent_Declare(
  Catch2
  GIT_REPOSITORY https://github.com/catchorg/Catch2.git
  GIT_TAG        v3.0.1 # or a later release
)
FetchContent_MakeAvailable(Catch2)
add_executable(test_v3 test_v3.cpp)
target_link_libraries(test_v3 PRIVATE Catch2::Catch2WithMain)

Recommended Usage

cmake can be configured to make tests runnable via make test or ctest

cmake_minimum_required(VERSION 3.18)
cmake_policy(SET CMP0048 NEW)
project(sample_test VERSION 1.0)
set(CMAKE_CXX_STANDARD 20)
include(CTest)
include(FetchContent)
FetchContent_Declare(
  Catch2
  GIT_REPOSITORY https://github.com/catchorg/Catch2.git
  GIT_TAG        v3.0.1 # or a later release
)
FetchContent_MakeAvailable(Catch2)
add_executable(test_v3 test_v3.cpp)
target_link_libraries(test_v3 PRIVATE Catch2::Catch2WithMain)
add_test(NAME TestV3 COMMAND test_v3)
Download CMakeLists.txt

Documentation

Comprehensive documentation can be found in Catch2's reference documentation.

The provided examples are helpful when getting started.

Test-driven development (TDD)

TDD is a methodology that emphasizes writing tests before the actual code

Helps ensure that code is tested thoroughly and avoids regressions

Repeat the cycle for each new piece of functionality

  1. Write a failing test that describes the desired functionality
  2. Write the minimum amount of code necessary to make the test pass
  3. Refactor the code to improve its structure and maintainability
  4. Make sure that all tests pass

Whenever you are tempted to type something into a print statement or a debugger expression, write it as a test instead.
– Martin Fowler

Questions
and feedback...