the Compartmented Robust Posix C++ Unit Test system

Chapter 1. crpcut philosophy

1.1. Why was crpcut written?
1.2. Why run tests individually in separate processes?
1.3. But doesn't forking all these short lived processes consume a lot of time, making crpcut very slow?
1.4. Why not throw exception to report errors? After all, that is the natural C++ error reporting mechanism.
1.5. Why terminate a test process on the first found error? Isn't it better to continue and see if there are more errors?
1.6. Why doesn't crpcut support parametrized tests?
1.7. How do I share a test setup that is expensive to initialize? I don't want to do it over for every test.
1.8. How do I pass state from one test to another, to chain them, if they run in isolated processes, or even in parallel?
1.9. But I really really really want to chain the result of tests. How do I pass state from one test to another?
1.10. Why the macros and template trickery, instead of a more normal C++ use?
1.11. Why require functionality that isn't even standardized yet, like variadic macros and decltype?

1.1.

Why was crpcut written?

Because, even though there are many C++ unit test systems out there, none covered all the desired properties, although many covered some:

  • Ease of writing tests. If it isn't easy, it won't be done, and most certainly won't be maintained.

  • Handle inadvertent crashes and non-progress (like infinite wait and infinite loops.)

  • Make it possible to fake system errors, such as a socket disconnected unexpectedly or disk full. The handling of these error situations are what defines program robustness, and they are extremely difficult to test without support from the test tool.

  • Can express dependencies between tests.

1.2.

Why run tests individually in separate processes?

Because it protects the tests from each other. One failed test cannot easily corrupt the environment for any other test.

It also makes it possible to handle inadvertent crashes and even infinite loops as any other failure.

1.3.

But doesn't forking all these short lived processes consume a lot of time, making crpcut very slow?

Not really. The overhead is very low, typically much less than a millisecond, so you would need a very long suite of very short tests for this to matter. Experience shows that build-time for a test program is typically much longer than the run-time, and the test programs are usually build-once/run-once.

In addition, crpcut allows you to run tests in parallel, which on a multi-core CPU can reduce run time considerably, especially if individual tests are time consuming.

1.4.

Why not throw exception to report errors? After all, that is the natural C++ error reporting mechanism.

It is a bad idea just because it is the natural C++ error reporting mechanism. A unit under test could accidentally prevent an error report from being captured. Consider the following class, that needs testing:


    class catchall
    {
    public:
      catchall(int n);
    };

    inline catchall::catchall(int n)
    {
      try {
        stubbed_function(n);
      }
      catch (...)
      {
         std::cerr << "something bad happened, but I saved the day" << std::endl;
      }
    }

Assume a macro THROW_ON_ERROR that behaves like an assert, except that it throws. It is used by the stub to verify that the caller does the right thing.


    int expected_value = 0;
    void stubbed_function(int n)
    {
      THROW_ON_ERROR(n != expected_value);
    }

Now we write a test for this program:


     TEST(construct_catchall_with_wrong_value)
     {
       expected_value = 3;
       catchall object(2);
     }

See what's wrong?

1.5.

Why terminate a test process on the first found error? Isn't it better to continue and see if there are more errors?

The thought is tempting, but if an early assertion is found to be false, how likely is the result from the rest of the code to be of any real value? In the worst case it will do something harmful. If the test is terminated immediately, it cannot do any further damage.

1.6.

Why doesn't crpcut support parametrized tests?

Parametrized tests, the way they are implemented in other unit test systems, are convenient, but also makes it a bit difficult to find what was in error and what wasn't.

The crpcut way of doing parametrized tests, is to express more or less the entire test functionality in a fixture using templates, and add several tests, each using different parameters for the test. Example:


    class parameter_base
    {
    protected:
      template <typename T1, typename T2>
      void my_test(T1 t1, T2 t2) {
        ASSERT_GT(t1, t2);
      }
    };

    TEST(gt_int_4_int_3, parameter_base)
    {
      my_test(4, 3);
    }

    TEST(gt_double_pi_int_3, parameter_base)
    {
      my_test(3.141592, 3);
    }

1.7.

How do I share a test setup that is expensive to initialize? I don't want to do it over for every test.

You initialize your setup once, outside the tests. It is probably a good idea to use a singleton, which you initialize from the main() function. The tests will get copies of the state when the processes are forked.

1.8.

How do I pass state from one test to another, to chain them, if they run in isolated processes, or even in parallel?

You really don't want to do that for a software unit test. If you were running a hardware production test, it might be interesting, in order to shorten the time to find defective units, but for a software unit test you want to pinpoint logical errors, and fix bugs. To do that effectively, you want to be able to run each test case individually.

1.9.

But I really really really want to chain the result of tests. How do I pass state from one test to another?

Sigh, if you insist on making like difficult for yourself, then so be it. What you do is that you set up a shared memory region from the main() function, and use that region to pass state. Do not forget to use the DEPENDS_ON(...) modifier to impose a strict order.

1.10.

Why the macros and template trickery, instead of a more normal C++ use?

It was not an easy decision, but the reason is simple. As ugly as the implementation is, the ease of use is phenomenal. You focus on writing test logic instead of writing boiler plate code to match the test engine's implementation.

1.11.

Why require functionality that isn't even standardized yet, like variadic macros and decltype?

Most compilers today already support the functionality, so it isn't too outrageous. The advantage gained in ease of writing test cases far outweighs the disadvantage.

[Note]Note
It is only the test code itself that must be compiled with support for the non-standard functionality. The units under test do not.