the Compartmented Robust Posix C++ Unit Test system
Test reports
Prev Chapter 3. Running tests Next

Test reports

The output from a test program is

By default, only the results from failed tests are listed, although the -v / --verbose command line flag changes that to include also the result from successful tests.

The result from an individual test includes:

In addition, failed tests include

A short explanation of the phase of failures:

phase

creating
The test failure occurred during the creation of the fixtures.
running
The test failure occurred during the execution of the test function body.
destroying
The test failure occurred during the destruction of the fixtures.
post_mortem
The test failure was detected by the main process after the test process had completed successfully
child
The test has spawned a child process, which has executed crpcut code. The entire process group is killed by crpcut

If the test result is directed to a file, only the result summary is shown on stdout (unless silenced by the -q / --quiet command line flag, in which case nothing at all is shown on stdout.)

Example program

     
     #include <crpcut.hpp>
     
     DEFINE_TEST_TAG(a_tag);
     DEFINE_TEST_TAG(another_tag);
     
     struct A
     {
       A() {}
       ~A() { abort(); }
       int func() const { return 3; }
       void do_something() const {}
     };
     
     TESTSUITE(basics)
     {
       TEST(construct, WITH_TEST_TAG(a_tag))
       {
         A *p = new A(); // leak
         INFO << "created an A, addr=" << p;
       }
     
       TEST(call_func, A, DEPENDS_ON(construct), WITH_TEST_TAG(a_tag))
       {
         ASSERT_TRUE(func() == 1);
       }
       TEST(destroy, A, DEPENDS_ON(construct), WITH_TEST_TAG(another_tag))
       {
         INFO << "running test body";
       }
     }
     
     TESTSUITE(toy, DEPENDS_ON(ALL_TESTS(basics)))
     {
       TEST(work)
       {
         A obj;
         obj.do_something();
       }
     }
     
     int main(int argc, char *argv[])
     {
       return crpcut::run(argc, argv);
     }

Human readable output

The example program shows all types of information in its end report when run with the command line flags --verbose --tags=/another_tag --xml=no: 


     PASSED!: basics::construct
     info---------------------------------------------------------------------------
     created an A, addr=0x8b0300
     ===============================================================================
     FAILED!: basics::call_func
     phase="running"  --------------------------------------------------------------
     samples/report_example.cpp:51
     ASSERT_TRUE(func() == 1)
       is evaluated as:
         3 == 1
     -------------------------------------------------------------------------------
     ===============================================================================
     FAILED?: basics::destroy
     info---------------------------------------------------------------------------
     running test body
     -------------------------------------------------------------------------------
     /tmp/crpcuti5XTwx/basics::destroy is not empty!
     phase="destroying"  -----------------------------------------------------------
     Died with core dump
     -------------------------------------------------------------------------------
     ===============================================================================
     Files remain under /tmp/crpcuti5XTwx
     The following tests were blocked from running:
       toy::work
     4 test cases selected
      tag             run  passed  failed
     !a_tag             2       1       1
     ?another_tag       1       0       1
     
     Total    :     Sum   Critical   Non-critical
     PASSED   :       1          1              0
     FAILED   :       2          1              1
     UNTESTED :       1

XML output

The same example program shows the XML formatted result shown below when run with the command line flags --verbose --tags=/another_tag --xml=yes: 


     <?xml version="1.0"?>
     
     <crpcut xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:noNamespaceSchemaLocation="http://crpcut.sourceforge.net/crpcut-1.6.0.xsd"
             starttime="2012-01-19T20:04:09Z"
             host="pteranodon"
             command="../test/report_example --xml=yes -verbose --tags=/another_tag">
       <test name="basics::construct" critical="true" result="PASSED">
         <log>
         <info>created an A, addr=0x8b0260</info>
         </log>
       </test>
       <test name="basics::call_func" critical="true" result="FAILED">
         <log>
           <violation phase="running">samples/report_example.cpp:51
     ASSERT_TRUE(func() == 1)
       is evaluated as:
         3 == 1</violation>
         </log>
       </test>
       <test name="basics::destroy" critical="false" result="FAILED">
         <log>
         <info>running test body</info>
           <violation phase="destroying" nonempty_dir="/tmp/crpcut6xQKNN/basics::destroy">Died with core dump</violation>
         </log>
       </test>
       <remaining_files nonempty_dir="/tmp/crpcut6xQKNN"/>
       <blocked_tests>
         <test name="toy::work"/>
       </blocked_tests>
       <tag_summary>
         <tag name="a_tag" passed="1" failed="1" critical="true"/>
         <tag name="another_tag" passed="0" failed="1" critical="false"/>
       </tag_summary>
       <statistics>
         <registered_test_cases>4</registered_test_cases>
         <selected_test_cases>4</selected_test_cases>
         <untested_test_cases>1</untested_test_cases>
         <run_test_cases>3</run_test_cases>
         <failed_test_cases>2</failed_test_cases>
         <failed_non_critical_test_cases>1</failed_non_critical_test_cases>
       </statistics>
     </crpcut>

Prev Up Next
The command line Home Debugging a test