I don't know why, but I love writing small test frameworks. The first serious one I wrote was for my bachelor thesis a couple of years ago, but I've used them as an exercise to play with languages many times before. This time, it was was different: test.h is a header-only testing framework for C that was born out of necessity.

Why do I find myself writing test frameworks?

For one, they're a nice way to put some stress on a language's features without having to invoke antediluvian horrors. Usually, test frameworks require some trickery with code generation, build-time conditionals or access to the interpreter's internals. It really depends on the language, but each has a more or less elegant way to obscure the boilerplate required to write a test.

In this case, I wasn't playing with C, but I actually needed a testing framework to write and verify a doubly-linked list. A few months ago (this post is way overdue) I did a job interview and my list skills came out a bit short to my liking, so I opened my editor and started doing some TDD. However, I didn't want to spend an unknown amount of time struggling to add an unknown testing framework to a build system before writing my first assertion. I just wanted a list.c file to contain both the list and the tests with a simple way of writing said tests.

Most test frameworks require that you either use CMake or find some obscure way to import them in a plain Makefile. I don't particularly like any of the two build systems, but let's just say I'm more than happy to write some recursive Makefiles to avoid doing anything else. So again, I didn't want to spend time figuring out how to integrate a test framework someone else wrote in a Makefile I didn't even have.

It turns out that you can write a simple test framework in C that fits in just a single test.h file, so as long as you paste it next to the file you're compiling, you just need to run:

ls # my-program.c test.h
gcc my-program.c -o my-program

Neat!

What does test.h offer?

test.h lets you write tests in plain C99 by declaring them with the macros Test and LongTest:

Test(popping_front_node_in_one_node_list_clears_it,
{
    List *list = List_new(10);
    list = List_pop_front(list);
    ExpectNull(list);
})

LongTest(pushing_back_and_popping_front_leaves_tail,
{
    List *list = List_new(10);
    List_push_back(list, 99);
    list = List_pop_front(list);

    AssertNotNull(list);
    ExpectEq(List_full_length(list), 1);
    ExpectEq(list->value, 99);

    TestTeardown({
        List_free(list);
    });
})

Both test macros start with their function name, a comma and their function body. That's the code between the { }. You should use Test when your tests don't create any side effects. For that case, LongTest is your friend, which requires that you define a TestTeardown at the end of the test body.

The TestTeardown, which Test automatically includes, macro defines the done label failed assertions will jump to and return the test result to the caller, so you'll need to add it at the end of a LongTest for it to work. Any code you add in the teardown will run independently of the test result, so if an early assertion fails and the rest of expectations don't run, the teardown code will run nonetheless.

Finally, you need to invoke your tests at the bottom of your file, as C doesn't allow the static constructor tricky C++ does. The macro RunTests builds a main function that runs all the tests you add to the array initialiser it takes:

RunTests({
    popping_front_node_in_one_node_list_clears_it,
    pushing_back_and_popping_front_leaves_tail,
})

And what's the output, you ask?

Starting tests: 2 tests
[0/1]: popping_front_node_in_one_node_list_clears_it... OK
[1/1]: pushing_and_popping_back_node_leaves_self... OK

Test checkers: expectations, assertions and ASSERTIONS!

You can find all types about checkers in their section in the README, but here's the gist:

You should use Expect when a check may fail but it's still safe to continue the test and catch as many errors as possible at once. You should use Assert when it's not safe to continue running a test after the check failed. Lastly, you should use ASSERT when it's not safe to continue running any more tests after the check failed.

As an example, you should bail out early from a test with Assert when you fail to construct the object you want to test. On the other hand, bailing out from the testing program is useful if you detect that it doesn't make sense to run any more tests as all will fail or that may break your computer. For example, you should ASSERT that a certain pathPrefix is not an empty string if you intend to run rm -rf $(pathPrefix)/* in your tests.

A couple of extra goodies

Since you may want to use test.h to test very basic code, you can define TEST_H_DISABLE_OUTPUT and TEST_H_DISABLE_STRING_TESTS to avoid including <stdio.h> and <string.h> respectively. If you define both symbols, test.h won't include any headers from the standard library. Isn't that neat?

Of course, disabling the test output will break your ability to know exactly which tests failed and why, but the program's retcode will give you some information for free, as test.h builds it in two parts:

  • The four least significant bits tell you the kind of checks that failed:
    • 0001 means that at least one test was empty.
    • 0010 means that at least one Expect check failed.
    • 0100 means that at least one Assert check failed.
    • 1000 means that at least one ASSERT check failed.
  • The higher bits store the index of the first test that failed (which may be the zeroth test).

So don't fear if you see weird return codes when running test.h, they're telling you where to look for errors!

What's missing in test.h?

Well, a lot! Let's be real: there's not much that you can add to a header-only testing framework in less than 300 lines. Still, the main goal of test.h is to be a simple testing framework, not an hyper-configurable beast. As it is right now, I consider it feature complete.

You could, in theory, hack in lots of cool things, like a way to pass argc and argv to the testing functions, but I think it would make the library much more complex than it should be for very little benefit. The three assertion types and the retcode encoding are advanced enough for this use case.

One thing that stands out as a shortcoming is that error reporting is too simple. In fact, when a check fails, it prints the line that caused the failure after expanding the test macro, which can be hard to read. For example, consider this test:

LongTest(oh_no_this_test_will_fail,
{
    List *const list = List_new(10);
    AssertNull(list);

    TestTeardown({
        List_free(list);
    });
})

If AssertNull is at line 289, you'll read the following:

Starting tests: 1 tests
[0/0]: oh_no_this_test_will_fail...
ASSERTION ERROR: `(list) == (((void *)0))` at list.c:289
FAILED!!!! oh_no_this_test_will_fail

It's not the most informative at first glance, but you can learn to understand the error and it gives you the exact line where it failed. At the very least, error messages are formatted in a way that's easy to parse!

Conclusion

Writing test.h and tweaking while writing this article has been a nice exercise. I have used it in a couple other places apart from the original list with great results!

While writing these last lines, I found that I wasn't the first to have the idea of writing a header-only testing suite (which should have been obvious from the start) so I added some lines at the end of the README to list them. All of those add more detail to the output than my test.h. I may take a look to get some ideas some time in the future ;-)

As a final treat, I've uploaded the list.c code where I originally used test.h. Happy testing and happy hacking!