Test reports

Test reports
	Chapter 3. Running tests

The output from a test program is

A (possibly empty) list of results from individual tests.
A (possibly empty) list of tests that were blocked from running due to unfulfilled dependencies or because they are disabled.
A (possibly empty) list of results for each tag used by tests in the run.
A result summary.

By default, only the results from failed tests are listed, although the -v / --verbose command line flag changes that to include also the result from successful tests.

The result from an individual test includes:

The name of the test
The result of the test, PASSED or FAILED.
The criticality of the test as ! (critical) or ? (non-critical)
log-items, each with the kind of log - stdout, stderr, fail and INFO.

In addition, failed tests include

the phase of failure.
if necessary, information where left behind files can be found
A description of the failure

A short explanation of the phase of failures:

phase

creating: The test failure occurred during the creation of the fixtures.
running: The test failure occurred during the execution of the test function body.
destroying: The test failure occurred during the destruction of the fixtures.
post_mortem: The test failure was detected by the main process after the test process had completed successfully
child: The test has spawned a child process, which has executed crpcut code. The entire process group is killed by crpcut

If the test result is directed to a file, only the result summary is shown on stdout (unless silenced by the -q / --quiet command line flag, in which case nothing at all is shown on stdout.)

Example program

     
     #include <crpcut.hpp>
     
     DEFINE_TEST_TAG(a_tag);
     DEFINE_TEST_TAG(another_tag);
     
     struct A
     {
       A() {}
       ~A() { abort(); }
       int func() const { return 3; }
       void do_something() const {}
     };
     
     TESTSUITE(basics)
     {
       DISABLED_TEST(will_not_run)
       {
         FAIL << "shall never ever run!";
       }
     
       TEST(construct, WITH_TEST_TAG(a_tag))
       {
         A *p = new A(); // leak
         INFO << "created an A, addr=" << p;
       }
     
       TEST(call_func, A, DEPENDS_ON(construct), WITH_TEST_TAG(a_tag))
       {
         ASSERT_TRUE(func() == 1);
       }
     
       TEST(destroy, A, DEPENDS_ON(construct), WITH_TEST_TAG(another_tag))
       {
         INFO << "running test body";
       }
     }
     
     TESTSUITE(toy, DEPENDS_ON(ALL_TESTS(basics)))
     {
       TEST(work)
       {
         A obj;
         obj.do_something();
       }
     }
     
     int main(int argc, char *argv[])
     {
       return crpcut::run(argc, argv);
     }

Human readable output

The example program shows all types of information in its end report when run with the command line flags --verbose --tags=/another_tag --xml=no:


     PASSED!: basics::construct
     info---------------------------------------------------------------------------
     samples/report_example.cpp:51
     created an A, addr=0x639860
     ===============================================================================
     FAILED!: basics::call_func
     phase="running"  --------------------------------------------------------------
     samples/report_example.cpp:56
     ASSERT_TRUE(func() == 1)
       is evaluated as:
         3 == 1
     -------------------------------------------------------------------------------
     ===============================================================================
     FAILED?: basics::destroy
     info---------------------------------------------------------------------------
     samples/report_example.cpp:61
     running test body
     -------------------------------------------------------------------------------
     /tmp/crpcutI4RXOk/basics::destroy is not empty!
     phase="destroying"  -----------------------------------------------------------
     samples/report_example.cpp:59
     Died with core dump
     -------------------------------------------------------------------------------
     ===============================================================================
     Files remain under /tmp/crpcutI4RXOk
     The following tests were blocked from running:
       -basics::will_not_run
       !toy::work
     5 test cases selected
      tag             run  passed  failed
     !a_tag             2       1       1
     ?another_tag       1       0       1
     
                    Sum   Critical   Non-critical
     PASSED   :       1          1              0
     FAILED   :       2          1              1
     UNTESTED :       2

The tests that are blocked from running are listed with an importance symbol. '-' for a disabled test, '?' for a non-critical test, and '!' for a critical test. The same importance symbol is used for the tag summary.

XML output

The same example program shows the XML formatted result shown below when run with the command line flags --verbose --tags=/another_tag --xml=yes:


     <?xml version="1.0"?>
     
     <crpcut xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:noNamespaceSchemaLocation="http://crpcut.sourceforge.net/crpcut-1.9.4.xsd"
             starttime="2016-03-07T16:39:37Z"
             host="pteranodon"
             command="../test/report_example -d /tmp/crpcuttxD9I6 --xml=yes --verbose --tags=/another_tag">
       <test name="basics::construct" critical="true" duration_us="10817" result="PASSED">
         <log>
         <info location="samples/report_example.cpp:51">created an A, addr=0x639a40</info>
         </log>
       </test>
       <test name="basics::call_func" critical="true" duration_us="293" result="FAILED">
         <log>
           <violation phase="running" location="samples/report_example.cpp:56">ASSERT_TRUE(func() == 1)
       is evaluated as:
         3 == 1</violation>
         </log>
       </test>
       <test name="basics::destroy" critical="false" duration_us="1543" result="FAILED">
         <log>
         <info location="samples/report_example.cpp:61">running test body</info>
           <violation phase="destroying" nonempty_dir="/tmp/crpcuttxD9I6/basics::destroy" location="samples/report_example.cpp:59">Died with core dump</violation>
         </log>
       </test>
       <remaining_files nonempty_dir="/tmp/crpcuttxD9I6"/>
       <blocked_tests>
         <test name="basics::will_not_run" importance="disabled"/>
         <test name="toy::work" importance="critical"/>
       </blocked_tests>
       <tag_summary>
         <tag name="a_tag" passed="1" failed="1" critical="true"/>
         <tag name="another_tag" passed="0" failed="1" critical="false"/>
       </tag_summary>
       <statistics>
         <registered_test_cases>5</registered_test_cases>
         <selected_test_cases>5</selected_test_cases>
         <untested_test_cases>2</untested_test_cases>
         <run_test_cases>3</run_test_cases>
         <failed_test_cases>2</failed_test_cases>
         <failed_non_critical_test_cases>1</failed_non_critical_test_cases>
       </statistics>
     </crpcut>