Test reports

Test reports
	Chapter 3. Running tests

The output from a test program is

A (possibly empty) list of results from individual tests.
A (possibly empty) list of tests that were blocked from running due to unfulfilled dependencies.
A (possibly empty) list of results for each tag used by tests in the run.
A result summary.

By default, only the results from failed tests are listed, although the -v / --verbose command line flag changes that to include also the result from successful tests.

The result from an individual test includes:

The name of the test
The result of the test, PASSED or FAILED.
The criticality of the test as ! (critical) or ? (non-critical)
log-items, each with the kind of log - stdout, stderr, fail and INFO.

In addition, failed tests include

the phase of failure.
if necessary, information where left behind files can be found
A description of the failure

A short explanation of the phase of failures:

phase

creating: The test failure occurred during the creation of the fixtures.
running: The test failure occurred during the execution of the test function body.
destroying: The test failure occurred during the destruction of the fixtures.
post_mortem: The test failure was detected by the main process after the test process had completed successfully
child: The test has spawned a child process, which has executed crpcut code. The entire process group is killed by crpcut

If the test result is directed to a file, only the result summary is shown on stdout (unless silenced by the -q / --quiet command line flag, in which case nothing at all is shown on stdout.)

Example program

     
     #include <crpcut.hpp>
     
     DEFINE_TEST_TAG(a_tag);
     DEFINE_TEST_TAG(another_tag);
     
     struct A
     {
       A() {}
       ~A() { abort(); }
       int func() const { return 3; }
       void do_something() const {}
     };
     
     TESTSUITE(basics)
     {
       TEST(construct, WITH_TEST_TAG(a_tag))
       {
         A *p = new A(); // leak
         INFO << "created an A, addr=" << p;
       }
     
       TEST(call_func, A, DEPENDS_ON(construct), WITH_TEST_TAG(a_tag))
       {
         ASSERT_TRUE(func() == 1);
       }
       TEST(destroy, A, DEPENDS_ON(construct), WITH_TEST_TAG(another_tag))
       {
         INFO << "running test body";
       }
     }
     
     TESTSUITE(toy, DEPENDS_ON(ALL_TESTS(basics)))
     {
       TEST(work)
       {
         A obj;
         obj.do_something();
       }
     }
     
     int main(int argc, char *argv[])
     {
       return crpcut::run(argc, argv);
     }

Human readable output

The example program shows all types of information in its end report when run with the command line flags --verbose --tags=/another_tag --xml=no:


     PASSED!: basics::construct
     info---------------------------------------------------------------------------
     created an A, addr=0x61f160
     ===============================================================================
     FAILED!: basics::call_func
     phase="running"  --------------------------------------------------------------
     samples/report_example.cpp:51
     ASSERT_TRUE(func() == 1)
       is evaluated as:
         3 == 1
     -------------------------------------------------------------------------------
     ===============================================================================
     FAILED?: basics::destroy
     info---------------------------------------------------------------------------
     running test body
     -------------------------------------------------------------------------------
     /tmp/crpcutwkiXNp/basics::destroy is not empty!!
     phase="destroying"  -----------------------------------------------------------
     Died with core dump
     -------------------------------------------------------------------------------
     ===============================================================================
     Files remain under /tmp/crpcutwkiXNp
     The following tests were blocked from running:
       toy::work
     4 test cases selected
      tag             run  passed  failed
     ?another_tag       1       0       1
     !a_tag             2       1       1
     
     Total    :     Sum   Critical   Non-critical
     PASSED   :       1          1              0
     FAILED   :       2          1              1
     UNTESTED :       1

XML output

The same example program shows the XML formatted result shown below when run with the command line flags --verbose --tags=/another_tag --xml=yes:


     <?xml version="1.0"?>
     
     <crpcut xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:noNamespaceSchemaLocation="http://crpcut.sourceforge.net/crpcut-1.5.0.xsd"
             starttime="2012-01-T16:29:09Z"
             host="pteranodon"
             command="../test/report_example --xml=yes -verbose --tags=/another_tag">
       <test name="basics::construct" critical="true" result="PASSED">
         <log>
         <info>created an A, addr=0x61f0c0</info>
         </log>
       </test>
       <test name="basics::call_func" critical="true" result="FAILED">
         <log>
           <violation phase="running">samples/report_example.cpp:51
     ASSERT_TRUE(func() == 1)
       is evaluated as:
         3 == 1</violation>
         </log>
       </test>
       <test name="basics::destroy" critical="false" result="FAILED">
         <log>
         <info>running test body</info>
           <violation phase="destroying" nonempty_dir="/tmp/crpcutUn3BKg/basics::destroy">Died with core dump</violation>
         </log>
       </test>
       <remaining_files nonempty_dir="/tmp/crpcutUn3BKg"/>
       <blocked_tests>
         <test name="toy::work"/>
       </blocked_tests>
       <tag_summary>
         <tag name="a_tag" passed="1" failed="1" critical="true"/>
         <tag name="another_tag" passed="0" failed="1" critical="false"/>
       </tag_summary>
       <statistics>
         <registered_test_cases>4</registered_test_cases>
         <selected_test_cases>4</selected_test_cases>
         <untested_test_cases>1</untested_test_cases>
         <run_test_cases>3</run_test_cases>
         <failed_test_cases>2</failed_test_cases>
         <failed_non_critical_test_cases>1</failed_non_critical_test_cases>
       </statistics>
     </crpcut>