Testing¶

While scanning modules, Maud will detect and register C++ unit tests. (This can be disabled by setting BUILD_TESTING = OFF.) Unit testing is based on GTest, and many basic concepts like suites of test cases are inherited whole. Many other aspects of writing tests are simplified. Instead of duplicating GTest’s documentation or explaining unit tests from the ground up, this documentation will assume familiarity and mostly describe the ways Maud’s usage differs.

Instead of defining test suites explicitly with classes, one test suite is produced for each C++ source which includes the special import declaration import test_. Each test suite is compiled into an executable target named test_.${STEM}.

In a suite source file, two macros are included in the predefines buffer (an explicit #include is unnecessary): test cases are defined with TEST_, and in a test case assertions are made with EXPECT_.

GTest is added to the include path, so explicit #include <gtest/gtest.h> is always available if necessary.

import test_;

TEST_(basic) {
  int three = 3, five = 5;
  EXPECT_(three == five);
  // ~/maud/.build/_maud/project_tests/unit testing/basics.cxx:12: Failure
  // Expected: three == five
  //   Actual:     3 vs 5
  EXPECT_(not three);
  // ~/maud/.build/_maud/project_tests/unit testing/basics.cxx:14: Failure
  // Expected: three
  //     to be false

  // EXPECT_(...) is an expression contextually convertible to bool
  if (not EXPECT_(&three != nullptr)) return;
  EXPECT_(&three != nullptr) or [](std::ostream &os) {
    // A lambda can hook expectation failure and add more context
  };

  // GMock's matchers are available
  EXPECT_("hello world" >>= HasSubstr("llo"));
}

// To check a test body against multiple values, parameterize with a range.
TEST_(parameterized, {111, 234}) {
  EXPECT_(parameter > 0);
}

// To instantiate the test body with multiple types, parameterize with a tuple.
TEST_(typed, 0, std::string("")) {
  EXPECT_(parameter + parameter == parameter);
}

Unit test API¶

TEST_(case_name, parameters...)¶ [source]

Defines and registers a test case with optional parameters.

Parameters:

case_name – The test case’s name
parameters – Parameters with which to parameterize the test body

If no parameters are provided, a single simple test case is defined.

TEST_(basic) {
  // assertions etc...
}

If parameters are provided, each is wrapped into a distinct test case using the same test body. In the scope of the test body, the parameter is declared as

Parameter const &parameter¶

If parameters are read from an initializer list or other range then this is analogous to a value parameterized test.

TEST_(value_parameterized, {2, 3, 47, 8191}) {
  EXPECT_(is_prime(parameter));
}

Parameters may also differ in type if they are read from a tuple, analogous to a type parameterized test.

TEST_(type_parameterized, 0, std::string("")) {
  EXPECT_(parameter + parameter == parameter);
}

Each parameter is printed and incorporated into the test case’s total name along with case_name and the suite’s name to make it accessible to filtering.

EXPECT_(condition...)¶ [source]

Checks its condition, producing a failure if it is falsy.

Parameters:

condition – An expression which is expected to be truthy.

To provide more information about a failed expectation, the condition will be printed as part of the failure. If the condition is a comparison, each argument will be printed.

int three = 3, five = 5;
EXPECT_(three == five);
// ~/maud/.build/_maud/project_tests/unit testing/basics.cxx:12: Failure
// Expected: three == five
//   Actual:     3 vs 5

EXPECT_(...) produces an expression rather than a statement. It is contextually convertible to bool, truthy iff the condition was truthy. If additional context needs to be added to a failed expectation, a lambda can be provided which will only be called if the expectation fails.

EXPECT_(&a == &b) or [&](auto &os) {
  os << "Extra context: " << a << " vs " << b;
};

Matchers can also be used to write an assertion with EXPECT_ through use of an overloaded operator>>=. (All matchers provided by GMock are exported by test_ and so are available in a test suite without an explicit #include.)

auto str  = "hello world";
EXPECT_(str >>= HasSubstr("boo"));
// ~/maud/.build/_maud/project_tests/unit testing/basics.cxx:21: Failure
// Expected: str has substring "boo"
// Argument was: "hello world"

template<typename Match, typename Description = DefaultDescription<Match>> class Matcher¶ [source]

Helper for constructing matchers from lambdas.

Matchers can be used with EXPECT_ using operator>>=.

For example, to define a matcher which checks whether a number is even:

Matcher IsEven = [](auto n, std::ostream &os) {
  return (n % 2) == 0;
};

To parameterize a matcher, define a function which returns a matcher with the parameters in closure:

auto IsDivisibleBy(auto divisor) {
  return Matcher{[=](auto n, std::ostream &os) {
    os << "where the remainder is " << (n % divisor);
    return (n % divisor) == 0;
  }};
}

On failed expectations, matchers output a description of the way matching failed. By default, this uses the type_name<T> of the match lambda (which is usually something unique but uninformative, like "$_1"). Description of the matcher can be customized with another lambda:

auto BarPlusBazEq(int n) {
  return Matcher{
    .match = [=](Foo f, std::ostream &os) { return f.bar() + f.baz() == n; },
    .description = [=](std::ostream &os, bool negated) {
      os << "bar() + baz()";
      os << (negated ? " does not equal " : " equals ") << n;
    },
  };
}

Frequently, a lambda is sufficient for constructing a custom matcher. However, it’s worth noting that defining a custom matcher class is not prohibitively complex.

template<typename T> std::string const type_name¶ [source]

A string representation of a type’s name.

By default this is a best effort demangling from type_info. This template can be specialized to override the default string.

template <>
std::string const type_name<Set<int>> = "Selection";

Custom `main()`¶

Each suite is linked to gtest_main. Since that defines main as a weak symbol, a custom main function can be written in a test suite and it will override gtest_main’s default.

To write a custom main function for all test executables, write an interface unit with export module test_:main; and that will be linked to each test executable instead of gtest_main.

Overriding `test_`¶

If it is preferable to override test_ entirely (for example to use a different test library like Catch2 instead of GTest), write an interface unit with export module test_ and define the cmake function maud_add_test:

maud_add_test(source_file out_target_name)

If defined, this function will be invoked on each source file which declares import test_, module test_, or any partition of it. If out_target_name is set to a target name, the source file will be attached to it and imports automatically processed as with other Maud targets. For example, if you would prefer to unit test with a minimal custom framework you could define your own module test_:

.test_.cxx

module;
#include "my_test_framework.hxx"
export module test_;
export using my_test_framework::expect_equals;
// ...

Then inject this into unit tests by defining maud_add_test:

test_.cmake

function(maud_add_test source_file out_target_name)
  cmake_path(GET source_file STEM name)
  set(${out_target_name} "test_.${name}" PARENT_SCOPE)

  # Create a test executable and register it
  add_executable(test_.${name})
  add_test(NAME test_.${name} COMMAND $<TARGET_FILE:test_.${name}>)

  # Link the unit test with test_.cxx:
  target_sources(
    test_.${name}
    PRIVATE FILE_SET module_providers TYPE CXX_MODULES
    BASE_DIRS ${CMAKE_SOURCE_DIR} FILES .test_.cxx
  )
endfunction()

Then this could be used in a unit test:

math.test.cxx

import test_;
int main() {
  expect_equals(1 + 2, 3);
}

Formatting test¶

By default, if clang-format is detected then a test will be added which asserts that files are formatted consistently:

$ ctest --build-config Debug --tests-regex formatted --output-on-failure
Test project ~/maud/.build
    Start 4: check.clang-formatted
1/1 Test #4: check.clang-formatted ............***Failed    0.07 sec
Clang-formating 16 files
~/maud/in2.cxx:15:42: error: code should be clang-formatted [-Wclang-format-violations]
export void compile_in2(std::istream &is,   std::ostream &os);
                                         ^

A fix target will also be added which formats files in place:

$ ninja fix.clang-format

Since the set of files which should be formatted is not necessarily identical to the set which should be compiled, the separate glob(MAUD_CXX_FORMATTED_SOURCES) configures which files clang-format will be applied to. Since different major versions of clang-format will frequently have different and backwards-incompatible behavior and parameter spaces, the target version must be specified in a .clang-format file:

# Versions: 18
BasedOnStyle: Google
ColumnLimit: 90

Testing¶

Unit test API¶

Custom main()¶

Overriding test_¶

Formatting test¶

Custom `main()`¶

Overriding `test_`¶