Test reports

Test reports
	Chapter 3. Running tests

The output from a test program is

A (possibly empty) list of results from individual tests.
A (possibly empty) list of tests that were blocked from running due to unfulfilled dependencies.
A result summary.

By default, only the results from failed tests are listed, although the -v command line flag changes that to include also the result from successful tests.

The result from an individual test includes:

The name of the test
The result of the test, PASSED or FAILED.
log-items, each with the kind of log - stdout, stderr and INFO.

In addition, failed tests include

the phase of failure discovery.
if necessary, information where left behind files can be found
A description of the failure

A short explanation of the phase of failure discovery:

phase

creating: The test failure was detected during the creation of the fixtures.
running: The test failure was detected during the execution of the test function body.
destroying: The test failure was detected during the destruction of the fixtures.
post_mortem: The test failure was detected by the main process after the test process died
child: The test has spawned a child process, which has executed crpcut code. The entire process group is killed by crpcut

If the test result is directed to a file, only the result summary is shown on stdout (unless silenced by the -q command line flag, in which case nothing at all is shown on stdout.

Human readable

Example: The following test program

     
     #include <crpcut.hpp>
     
     struct A
     {
       A() {}
       ~A() { abort(); }
       void do_something() const {}
     };
     
     TESTSUITE(basics)
     {
       TEST(construct)
       {
         A *p = new A(); // leak
         INFO << "created an A, addr=" << p;
       }
     
       TEST(destroy, DEPENDS_ON(construct))
       {
         A obj;
       }
     }
     
     TESTSUITE(toy, DEPENDS_ON(ALL_TESTS(basics)))
     {
       TEST(work)
       {
         A obj;
         obj.do_something();
       }
     }
     
     int main(int argc, char *argv[])
     {
       return crpcut::run(argc, argv);
     }

shows all types of information in its end report when run with the -v command line flag.


     PASSED: basics::construct
     info---------------------------------------------------------------------------
     created an A, addr=0x2b244bb682d0
     ===============================================================================
     FAILED: basics::destroy
     /tmp/crpcutotscwd/basics::destroy is not empty!!
     phase="running"  --------------------------------------------------------------
     Died with core dump
     -------------------------------------------------------------------------------
     ===============================================================================
     Files remain under /tmp/crpcutotscwd
     The following tests were blocked from running:
       toy::work
     Total 3 test cases selected
     UNTESTED : 1
     PASSED   : 1
     FAILED   : 1

XML

Example: The following test program

     
     #include <crpcut.hpp>
     
     struct A
     {
       A() {}
       ~A() { abort(); }
       void do_something() const {}
     };
     
     TESTSUITE(basics)
     {
       TEST(construct)
       {
         A *p = new A(); // leak
         INFO << "created an A, addr=" << p;
       }
     
       TEST(destroy, DEPENDS_ON(construct))
       {
         A obj;
       }
     }
     
     TESTSUITE(toy, DEPENDS_ON(ALL_TESTS(basics)))
     {
       TEST(work)
       {
         A obj;
         obj.do_something();
       }
     }
     
     int main(int argc, char *argv[])
     {
       return crpcut::run(argc, argv);
     }

shows all types of information in its end report when run with the -v command line flag.


     <?xml version="1.0"?>
     
     <crpcut xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="crpcut.xsd" starttime="2009-12-15T19:23:48Z" host="pteranodon" command="/var/tmp/build/test/report_example -v -x">
       <test name="basics::construct" result="PASSED">
         <log>
         <info>created an A, addr=0x2b5fcb3eaae0</info>
         </log>
       </test>
       <test name="basics::destroy" result="FAILED">
         <log>
           <violation phase="running" nonempty_dir="/tmp/crpcut5Mwgtf/basics::destroy">Died with core dump</violation>
         </log>
       </test>
       <remaining_files nonempty_dir="/tmp/crpcut5Mwgtf"/>
       <blocked_tests>
         <test name="toy::work"/>
       </blocked_tests>
       <statistics>
         <registered_test_cases>3</registered_test_cases>
         <selected_test_cases>3</selected_test_cases>
         <untested_test_cases>1</untested_test_cases>
         <run_test_cases>2</run_test_cases>
         <failed_test_cases>1</failed_test_cases>
       </statistics>
     </crpcut>