Chapter 3. Grouping tests

In crpcut a testsuite is a list of tests and enclosed testsuites. At the out most level, tests are included in the unnamed testsuite. With testsuites you can describe a hierarchy of tests, since testsuites can contain other testsuites.

Testsuites are declared using the TESTSUITE(name, ...) macro. A testsuite is a C++ namespace with some decorations attached. For example it is possible to express dependencies on testsuites (i.e. the requirement that all tests in a testsuite pass,) and it is possible for testsuites to depend on tests and other testsuites.

	Tip
	Experience has shown that it is a good idea to let testsuites depend on testsuites, and tests in testsuites depend on other tests in the same testsuite

Consider the simple symtable class from earlier:

     #include <map>
     #include <string>
     #include <cassert>
     
     class symtable
     {
     public:
       void add(const char *name, int val)
       {
         assert(name);
         table[name] = val;
       }
       int lookup(const char *name)
       {
         assert(name);
         return table.at(name);
       }
     private:
       std::map<std::string, int> table;
     };

The following longish test program follows the recommended form for testsuites with dependencies:

     #include "symtable.hpp"
     #include <crpcut.hpp>
     
     TESTSUITE(construct_and_destroy)
     {
       TEST(construct)
       {
         (void)new symtable; // intentional leak
       }
     
       TEST(destroy, DEPENDS_ON(construct))
       {
         symtable s;
       }
     }
     
     TESTSUITE(normal_access, DEPENDS_ON(ALL_TESTS(construct_and_destroy)))
     {
       TEST(insert_one)
       {
         symtable s;
         s.add("one", 1);
       }
       TEST(insert_several, DEPENDS_ON(insert_one))
       {
         symtable s;
         s.add("one", 1);
         s.add("two", 2);
         s.add("three", 3);
       }
       TEST(lookup, DEPENDS_ON(insert_several))
       {
         symtable s;
         s.add("one", 1);
         s.add("two", 2);
         int v = s.lookup("one");
         ASSERT_EQ(v, 1);
         v = s.lookup("two");
         ASSERT_EQ(v, 2);
       }
     }
     
     TESTSUITE(abnormal, DEPENDS_ON(ALL_TESTS(normal_access)))
     {
       TEST(lookup_nonexisting, EXPECT_EXCEPTION(std::out_of_range))
        {
          symtable s;
          s.add("one", 1);
          s.lookup("two");
        }
     
       TEST(add_null, EXPECT_SIGNAL_DEATH(SIGABRT), NO_CORE_FILE)
       {
         symtable s;
         s.add(0, 1);
       }
     
       TEST(lookup_null, EXPECT_SIGNAL_DEATH(SIGABRT), NO_CORE_FILE)
       {
         symtable s;
         s.add("one", 1);
         s.lookup(0);
       }
     }
     
     int main(int argc, char *argv[])
     {
       return crpcut::run(argc, argv);
     }

Testsuites are also useful when there is a desire to run a subset of the tests. Running the above test program with "--verbose normal_access" yields:

     PASSED!: normal_access::insert_one
     ===============================================================================
     PASSED!: normal_access::insert_several
     ===============================================================================
     PASSED!: normal_access::lookup
     ===============================================================================
     3 test cases selected
     
                    Sum   Critical   Non-critical
     PASSED   :       3          3              0

(the -v / --verbose command line flag makes crpcut list output also from the tests that succeeds.)

	Note
	The command line flag -n / --nodeps is not necessary since dependencies are only calculated on tests selected to run, in this case the tests in the testsuite normal_access.

A tag in a crpcut test program is a group identity that can be attached to a test. A test can either be untagged (the default) or have one tag, never more.

In older C++ tags must be defined with the DEFINE_TEST_TAG(tagname) macro before they can be used. It makes a tag as a C++ type with some decorations, and from there it follows that a tag must be visible at the site of use.

	Tip
	Compile your test program as C++11, then DEFINE_TEST_TAG(tagname) becomes unnecessary, and tags have a freer form and can be anything the preprocessor can parse.

To attach a tag to a test, include the WITH_TEST_TAG(tagname) macro in the modifiers list.

Revisit the simple symtable again:

     #include <map>
     #include <string>
     #include <cassert>
     
     class symtable
     {
     public:
       void add(const char *name, int val)
       {
         assert(name);
         table[name] = val;
       }
       int lookup(const char *name)
       {
         assert(name);
         return table.at(name);
       }
     private:
       std::map<std::string, int> table;
     };

Below is a simple test program showing the use of tags. It adds a test for the new requirement that an overwrite shall throw:

     #include "symtable.hpp"
     #include <crpcut.hpp>
     
     DEFINE_TEST_TAG(insert);
     DEFINE_TEST_TAG(lookup);
     
     TESTSUITE(construct_and_destroy)
     {
       TEST(construct)
       {
         (void)new symtable; // intentional leak
       }
     
       TEST(destroy, DEPENDS_ON(construct))
       {
         symtable s;
       }
     }
     
     TESTSUITE(normal_access, DEPENDS_ON(ALL_TESTS(construct_and_destroy)))
     {
       TEST(insert_one, WITH_TEST_TAG(insert))
       {
         symtable s;
         s.add("one", 1);
       }
       TEST(insert_several, WITH_TEST_TAG(insert),
                            DEPENDS_ON(insert_one))
       {
         symtable s;
         s.add("one", 1);
         s.add("two", 2);
         s.add("three", 3);
       }
       TEST(lookup, WITH_TEST_TAG(lookup),
                    DEPENDS_ON(insert_several))
       {
         symtable s;
         s.add("one", 1);
         s.add("two", 2);
         int v = s.lookup("one");
         ASSERT_EQ(v, 1);
         v = s.lookup("two");
         ASSERT_EQ(v, 2);
       }
     
     }
     
     TESTSUITE(abnormal, DEPENDS_ON(ALL_TESTS(normal_access)))
     {
       TEST(lookup_nonexisting, WITH_TEST_TAG(lookup),
                                EXPECT_EXCEPTION(std::out_of_range))
        {
          symtable s;
          s.add("one", 1);
          s.lookup("two");
        }
     
       TEST(add_null, WITH_TEST_TAG(insert),
                      EXPECT_SIGNAL_DEATH(SIGABRT),
                      NO_CORE_FILE)
       {
         symtable s;
         s.add(0, 1);
       }
     
       TEST(lookup_null, WITH_TEST_TAG(lookup),
                         EXPECT_SIGNAL_DEATH(SIGABRT),
                         NO_CORE_FILE)
       {
         symtable s;
         s.add("one", 1);
         s.lookup(0);
       }
     
       TEST(overwrite_throws, WITH_TEST_TAG(insert))
       {
         symtable s;
         s.add("one", 1);
         ASSERT_THROW(s.add("one", 1), std::overflow_error);
       }
     }
     
     int main(int argc, char *argv[])
     {
       return crpcut::run(argc, argv);
     }

Running this test program with the command line flag -L / --list-tags shows the list of all tags:

     insert
     lookup

Running the test program with --tags=/insert selects all tests for running and defines the tests tagged insert as non-critical. The result is:

     FAILED?: abnormal::overwrite_throws
     phase="running"  --------------------------------------------------------------
     samples/tag-example.cpp:106
     ASSERT_THROW(s.add("one", 1), std::overflow_error)
       Did not throw
     -------------------------------------------------------------------------------
     ===============================================================================
     9 test cases selected
      tag        run  passed  failed
     ?insert       4       3       1
     !lookup       3       3       0
     
                    Sum   Critical   Non-critical
     PASSED   :       8          5              3
     FAILED   :       1          0              1

The question mark in FAILED? indicates that the test that failed is non-critical.

Likewise, in the list of tags with passed/failed statistics for each, every line begins with an exclamation mark for critical tests, or a question mark for non-critical tests.

Tags can also be used to select which tests to run, and can be combined with name matching. Running the same program again with --verbose --tags=-lookup/insert construct_and_destroy abnormal selects only the tests in the testsuite construct_and_destroy and abnormal that do not have the tag lookup (the minus is a negative selection) and defines tests with the tag insert as non-critical. The result is:

     PASSED!: construct_and_destroy::construct
     ===============================================================================
     PASSED!: construct_and_destroy::destroy
     ===============================================================================
     PASSED?: abnormal::add_null
     stderr-------------------------------------------------------------------------
     tag-example: samples/symtable.hpp:36: void symtable::add(const char*, int): Assertion `name' failed.
     
     ===============================================================================
     FAILED?: abnormal::overwrite_throws
     phase="running"  --------------------------------------------------------------
     samples/tag-example.cpp:106
     ASSERT_THROW(s.add("one", 1), std::overflow_error)
       Did not throw
     -------------------------------------------------------------------------------
     ===============================================================================
     4 test cases selected
      tag        run  passed  failed
     ?insert       2       1       1
     
                    Sum   Critical   Non-critical
     PASSED   :       3          2              1
     FAILED   :       1          0              1