crpcut-0.8.2 (a.k.a 1.0-beta3) - The Compartmented Robust Posix C++ Unit Tester
|
crpcut (pronounced "crap cut") is the
Compartmented Robust Posix C++ Unit Tester
With crpcut it is easy to write tests that other unit-test frame works cannot
manage.
|
|
Introductory example
An example testing parts of std::string:
#include <crpcut.hpp>
#include <string>
struct apastr // fixture for mosts tests
{
apastr() : s("apa") {}
std::string s;
};
TESTSUITE(basics)
{
TEST(default_constr_and_destr)
{
std::string s;
ASSERT_TRUE(s.empty());
ASSERT_EQ(s.length(), 0);
}
TEST(constr_from_char_array, apastr,
DEPENDS_ON(default_constr_and_destr))
{
ASSERT_EQ(s.length(), 3UL);
}
TEST(at, apastr,
DEPENDS_ON(default_constr_and_destr))
{
ASSERT_EQ(s.at(1), 'p');
}
}
TESTSUITE(errors, DEPENDS_ON(ALL_TESTS(basics)))
{
TEST(out_of_mem, EXPECT_EXCEPTION(std::bad_alloc))
{
crpcut::heap::set_limit(crpcut::heap::allocated_bytes() + 2);
std::string s("apa");
}
TEST(at_out_of_range, apastr,
EXPECT_EXCEPTION(std::out_of_range))
{
s.at(4);
}
TEST(index_oper_out_of_range, apastr,
EXPECT_SIGNAL_DEATH(SIGABRT),
NO_CORE_FILE)
{
s[4];
}
}
int main(int argc, char *argv[])
{
return crpcut::test_case_factory::run_test(argc, argv);
}
Similar tests benefits from being grouped into test-suites. Test-suites
can depend on other test-suites, meaning that the contained tests will only
run if all tests it depends on have completed successfully. Results
can be validated using a number of ASSERT macros. Tests can use
fixtures to express common contents. Tests can be expected to exit by
exception, or die. Expectations that are not met are errors. Messages on
stderr and stdout are collected, and included in the result log.
Why crpcut
Most importantly, it must be easy to write tests. With crpcut, you focus
on your test structure and test logic, not on the limits imposed by
your test environment.
With crpcut, every test case runs in its own process and its own working
directory. If a test case fails, the process terminates immediately, before
it does further harm. This means that every test case starts from a clean
slate, unaffected by other tests. This is the compartmentalization.
It also means that the test suite continues, even if a test crashes.
You can set deadlines for test cases, and if the allowed time is seriously
overdrawn, the test case process is killed. These two make up the robustness
part.
You can define dependencies between test cases and between test suites, so that
if a fundamental tests fails, the tests that are based on the fundamental
functionality will not even be run.
The crpcut main process does not have any dynamic memory allocated at the
time a test case process is started, so you can run crpcut using a memory
test tool, such as valgrind,
and if there is memory allocated when the test
case process terminates, you can be assured that you have found a memory leak
in your test.
If you have a multi-core CPU, it may be beneficial to run several test cases
in parallel. crpcut allows that.
If there are files left in the test process' working directory after the test
case process has terminated, the test case is considered failed. The working
directory is left untouched by crpcut, for you to examine.
Competition
You may want to check out the competition and decide which system is best for
you. Some examples:
crpcut links
The current version of crpcut is 0.8.2 (a.k.a. 1.0-beta-3)
Latest version of this document
Sourceforge project page
Sourceforge download page
GIT repo at git://crpcut.git.sourceforge.net/gitroot/crpcut/crpcut
Support mailing list
bug-tracker
License
/*
* Copyright 2009 Bjorn Fahller <bjorn@fahller.se>
* All rights reserved
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
Requirements
crpcut relies on a few Posix functions, and three extensions to C++ (as defined
by the 1998 standard and the 2003 corrigendum.)
The C++ extensions are:
| extension |
|
where defined |
| variadic macros |
|
C99/c++0x |
| decltype |
|
c++0x (typeof, which in current gcc is similar enough,
is tried if decltype isn't availble) |
| _Exit() |
|
C99/c++0x |
All will become standardized for C++ soon, and are since some time supported
by compilers like g++ and
Intel's C++ compiler
ICC.
The required Posix functions are:
| Standard |
|
Requirement |
| POSIX.1 |
|
EEXIST
EINTR |
| POSIX.1-2001 |
|
alloca()
chdir()
** clock_gettime(CLOCK_MONOTONIC)
** clock_gettime(CLOCK_PROCESS_CPUTIME_ID)
close()
closedir()
dup2()
++ epoll_create()
++ epoll_ctl()
++ epoll_wait()
fork()
getcwd()
gethostname()
** getitimer(ITIMER_REAL)
** getitimer(ITIMER_VIRTUAL)
kill()
mkdir()
open()
opendir()
pipe()
read()
readdir_r()
-- regcomp()
-- regexec()
-- regerror()
-- regfree()
rmdir()
++ select()
** setitimer(ITIMER_REAL)
** setitimer(ITIMER_VIRTUAL)
setrlimit(RLIMIT_CORE)
setrlimit(RLIMIT_CPU)
waitid()
write() |
| POSIX.1-2008 |
|
mkdtemp() |
|
|
|
| ** |
|
clock_gettime() will be used if available.
*itimer() will be used as a fallback. |
| ++ |
|
epoll_* will be used if available. select()
will be used as a fallback. |
| -- |
|
reg* will be used only when matching strings with
regular expressions. |
|
|
On OS-X mach_absolute_time() and
mach_timebase_info() will be used for realtime measurements. |
This should work for most Linux systems. It is also likely to work for
OS-X, but it has not yet
been verified.
Build and Install
crpcut uses CMake to manage builds and
installs.
First obtain the crpcut sources, either a release from the
sourceforge
download
page, or from the git repository
git://crpcut.git.sourceforge.net/gitroot/crpcut/crpcut
If you intend to build a binary lib for distribution, it is recommended
to build on a host which has valgrind
installed. crpcut's heap management adds valgrind special instructions, if
valgrind.h is found, which helps track memory leaks. Those special
instructions are no-ops if valgrind
is not used or available.
In a build directory (can be the same as the source directory)
> cmake path
Where path is the path to the crpcut source directory.
Should you want to use a different install directory than the
CMake default, type instead
> cmake path -DCMAKE_INSTALL_PREFIX=<desired_install_directory>
If you want to use another compiler than your default, for example
Intel's C++ compiler
ICC,
add CXX=compilername before the cmake command. An example
for ICC
becomes
> CXX=icpc cmake ...
crpcut supports
google-mock
automatically, but should you want to include test cases for the built-in self
test, that verifies the
google-mock interaction,
add
-DWITH_GOOGLE_MOCK=yes
to the cmake line. If
google-mock is not
installed in a directory that is reachable by you C++ compiler and
linker, without additional flags, provide
cmake with the directory of
google-mock using:
-DGOOGLE_MOCK_DIR=path
where path is the exact same as used in the --prefix=path option for
configure in
google-mock.
Once cmake has completed, build crpcut
by typing:
> make
Once built, you may want to check that it works correctly. Do that by
running first builting the example code as a test program:
> make testprog
Then run the selftest itself:
> make selftest
It takes about 30 seconds to run through all tests. It verifies that the test
cases under test-src completes with the expected result for a large set
of command-line switches.
You can now install crpcut by typing:
> make install
The file install_manifest.txt is created and includes the full path
of all installed files.
Writing tests
Test cases are defined using the TEST() macro as:
TEST(name_of_testcase (,(fixture|modifier))*)
{
... code
}
Both fixtures and
modifiers
are explained below.
If several tests are related, they can be grouped into a test suite using the
TESTSUITE() macro. The macro is used to name a testsuite, and if
desired also use the DEPENDS_ON
modifier to define dependencies for all tests
in a testsuite. For a large range of tests, it may be a good idea
to nest test suites. A nested testsuite inherits all dependencies of its
enclosing testsuite. Test cases can be written in several files and linked to
one main program. If several files define the same test-suite name, the tests
will be joined it the same suite. Beware of dependencies with testsuites that
span several files; the dependencies are file-local. The crpcut_ name
prefix is reserved for crpcut.
In tests, assert correctness using the assertion macros.
The unary assertions are:
| assert |
|
condition |
| ASSERT_TRUE(expr) |
|
Fails iff expr evaluates to
false
|
| ASSERT_FALSE(expr) |
|
Fails iff expr evaluates to
true
|
| ASSERT_NO_THROW(expr) |
|
Fails iff expr throws
anything |
The binary assertions are:
| assert |
|
condition |
| ASSERT_EQ(a, b) |
|
Fails iff the expression (a == b)
evaluates to false
|
| ASSERT_NE(a, b) |
|
Fails iff the expression (a != b)
evaluates to false
|
| ASSERT_GT(a, b) |
|
Fails iff the expression (a > b)
evaluates to false
|
| ASSERT_GE(a, b) |
|
Fails iff the expression (a >= b)
evaluates to false
|
| ASSERT_LT(a, b) |
|
Fails iff the expression (a < b)
evaluates to false
|
| ASSERT_LE(a, b) |
|
Fails iff the expression (a <= b)
evaluates to false
|
| ASSERT_THROW(expr, exc_type) |
|
Fails iff the expression
expr does not throw
an exception of type
exc_type. Use
... for
exc_type if any
exception is good. |
There is also an N-ary assert macro:
| assert |
|
condition |
| ASSERT_PRED(predicate, ...) |
|
Fails iff the expression
predicate(...) evaluates
to false. predicate may
be anything that can act as a
function accepting the rest of
the parameters, and returns a value
comparable to false. |
If an assertion fails, the value of the parameters is included in the failure
log (as output streamed, if possible, or as hex-dump otherwise.) All assert
macros evaluates each parameter value exactly once, so providing parameters with
side effects is not a problem. The order of evaluation of the parameters is
undefined, however.
ASSERT_PRED has the advantage that it gives you all parameter values.
For example, the two tests below are logically equal:
const char *p = afunc();
ASSERT_TRUE(std::equal_to<std::string>()("expected", p));
and
const char *p = afunc();
ASSERT_PRED(std::equal_to<std::string>(), "expected", p);
However, of the two above, the latter will give much better violation
information, should the test fail. Compare:
ASSERT_TRUE(std::equal_to<std::string>()("expected", p))
where std::equal_to<std::string>()("expected", p) = false
and
ASSERT_PRED(std::equal_to<std::string>()("expected", p))
param1 = expected
param2 = otherval
If you define your own predicates (see the
section below) you can define an output stream operator for it, to make it
express the nature of the test even better.
In addition to the assertions, two output streamers are available. Both
are used as normal std::ostream objects.
| streamer |
|
action |
| FAIL |
|
Immediately terminate the test case as a failure with the
streamed message. |
| INFO |
|
Add the streamed message to the test-case log, but do
not fail it. |
Each FAIL or INFO output statement is a complete log
message. There is no need to add line-breaks or other separators.
An example:
TEST(fail_immediately)
{
int n = random();
INFO << "random value is " << n;
FAIL << "Goodbye cruel world";
}
Fixtures
If several test cases share the same test setup, test fixtures can be written.
A fixture is just a class or struct, with the desired
information. The default constructor is expected to fill in the desired
information, and the destructor to clean up afterwards. Several fixtures can
be combined. The fixtures are inherited by the test case.
Small example:
TESTSUITE(string_length)
{
class fixture1
{
protected:
fixture1() : i(3) {}
int i;
};
struct fixture2
{
std::string msg;
fixture2() : msg("cat");
};
TEST(check_length, fixture1, fixture2)
{
ASSERT_EQ(msg.length(), i);
}
TEST(check_c_length, fixture1, fixture2)
{
ASSERT_EQ(std::strlen(msg.c_str()), i);
}
}
Modifiers
In addition to using fixtures and asserts, the expected behaviour of
test cases can be altered using modifiers. Modifiers are listed together with
the fixtures.
The defined test case modifiers are:
|
|
|
| NO_CORE_FILE |
|
Make sure the test doesn't produce a core
file, no matter how it crashes. Useful when
testing that an assert() works as expected.
|
| EXPECT_EXIT(code) |
|
For the test to succeed, it must exit with the
supplied exit code. If any exit is OK, use
ANY_CODE for code. |
| EXPECT_SIGNAL_DEATH(code) |
|
For the test to succeed, it must
terminate on the supplied signal number.
If any signal number is OK, use
ANY_CODE for code.
|
| EXPECT_EXCEPTION(type) |
|
For the test to succeed, it must exit by
throwing an instance of the provided type.
If any exception is good, use
... for
type. |
| DEADLINE_CPU_MS(duration_ms) |
|
For the test to succeed it must run
to completion before consuming
duration_ms milliseconds
CPU-time. If the time consumed is
vastly more, the test process will
be killed (uses setrlimit()
with RLIMIT_CPU, and
clock_gettime() with
CLOCK_PROCESS_CPUTIME_ID.)
|
| DEADLINE_REALTIME_MS(duration_ms) |
|
For the test to succeed it
must run to completion before
consuming duration_ms
milliseconds on the
rate-monotonic clock. If the
time consumed is vastly more,
the crpcut engine will kill it
using kill() with
signal SIGKILL. Time
is measured using
clock_gettime() with
CLOCK_MONOTONIC. |
| DEPENDS_ON(...) |
|
... is a list of test cases, or
ALL_TESTS(ns). Before the test can
run, all tests in the list, or all tests in
TESTSUITE(ns) must have finished
successfully. |
Disbaled tests
If, for whatever reason, you have tests that you currently don't want to
run, but you intend for them to be included later, define them using
DISABLED_TEST() instead of TEST(). Test cases defined with
DISABLED_TEST() are compiled, preventing code-rot, but will never be
a candidate for running. It is not possible to state dependency on a disabled
test.
Comparing strings
crpcut currently provides two ways of comparing strings:
collation
Collation is comparing strings sort order in a locale. It is possible to mix
both std::string and C strings in collations. It is also possible to
collate wchar_t strings, providing the locale supports it.
The basic syntax is:
ASSERT_TRUE(crpcut::collate("aaa") op "aaaa")
where op is one of <, <=, ==, !=,
> and >=.
Note that == and != are some of a misnomer. Collation is
about sort order in a locale, so == means the strings are considered
unordered compared to one another, but that doesn't necessarily mean that
they are equal. In some locales, for example, an umlaut is a decoration
(in terms of collation) whereas in other locales the exact same umlaut
makes a letter with a different place in the alphabet.
To select which locale to work in, add the locale as the second parameter.
ASSERT_TRUE(crpcut::collate("aaa", std::locale("sv_SE")) op s);
If the locale cannot be constructed, the assertion fails.
Example:
TEST(de_sv_collation)
{
static const char s1[] = "öz";
static const char s2[] = "zö";
ASSERT_TRUE(crpcut::collate(s1, std::locale("de_DE")) < s2);
ASSERT_TRUE(crpcut::collate(s1, std::locale("sv_SE")) > s2);
}
You can also convert the strings to uppercase or lowercase before testing the
collation order. This is done using the templated variant of collate.
| template |
|
meaning |
| crpcut::collate<crpcut::uppercase>() |
|
translate reference string and compared string to upper case, in the
locale, before comparing collation order. |
| crpcut::collate<crpcut::lowercase>() |
|
translate reference string and compared string to lower case,
in the locale, before comparing collation order. |
| crpcut::collate<crpcut::verbatim>() |
|
synonymous with the non-templated version |
regular expression
Strings can be matched against regular expressions using ASSERT_PRED
with the match function template instantiated using regex:
ASSERT_PRED(crpcut::match<crpcut::regex>(pattern, flags...), string);
The pattern and the string can be either of std::string or C string.
There are three flags available:
| flag |
|
meaning |
| crpcut::regex::i |
|
Ignore case when matching. Equal to
REG_ICASE in regcomp(). Note
that this does not take any locale into
consideration. |
| crpcut::regex::e |
|
Use extended regular expressions. Equal to
REG_EXTENDED in regcomp()
|
| crpcut::regex::m |
|
Pattern is multi line. Equal to
REG_NEWLINE in regcomp()
|
The flags defaults to not set. You can set one, two or all three, as separate
trailing parameters.
Be careful with the C/C++ rules for escapes when a regular expression
requires \.
Example:
TEST(looks_like_ipaddr)
{
using crpcut::match;
using crpcut::regex;
const char *ipaddr = afunc();
static const char pattern[] = "^([[:digit:]]{1,3}\\.){3}[[:digit:]]{1,3}$";
ASSERT_PRED(match<regex>(pattern, regex::e), ipaddr);
}
Comparing floating point values
Comparing floating point values for equality is tricky. Due to rounding errors,
you do normally not want to check that a computed value is exactly equal to
an expected value, but rather that the computed value is close to the
expected, with some chosen precision.
crpcut provides three ways of expressing that precision, with their
advantages and disadvantages.
|
|
|
| relative_diff(diff)(a, b) |
|
Check that 2*|a-b|/|a+b|<=diff.
E.g. if diff=0.01, the values a and b
must not differ by more than 1%. The
main disadvantage of
relative_diff is when the
expected values are close to 0. |
| abs_diff(diff)(a, b) |
|
Check that |a-b|<=diff. This works
well for numbers near 0, but for large
numbers it quickly becomes
uninteresting. |
| ulps_diff(diff, inf)(a, b) |
|
Check that the number of
possible floating point
values between a and
b is not greater
than diff.
inf can be one of
include_inf or
exclude_inf and
controls whether infinity
is a valid number (one
representation away from max)
or not. It defaults to
exclude_inf. See
Cygnus information on
comparing floating point numbers, and also David Goldberg's What Every Computer Scientists Should Know About Floating-Point Arithmetic.
|
Floating point comparisons are made using ASSERT_PRED with the
match function template like this:
ASSERT_PRED(crpcut::match<crpcut::abs_diff>(0.0001), a, b);
The two values must all have exactly the same floating point type. For
relative_diff and abs_diff the precision must also be
of the same type as the compared values. For ulps_diff the
precision is an unsigned integer. ulps_diff is not available for
long double.
It is worth pointing out that arithmetics with long double might
work poorly when run under valgrind,
since it (at least in version 3.4.1 and older) never makes floating point
calculations with greater precision than
64 bits (double.)
Using google-mock with crpcut
When using google-mock
in crpcut tests, the order of the #include preprocessor directives
are important:
#include <gmock/gmock.h> // Always gmock/gmock.h before crpcut.hpp
#include <crpcut.hpp> // since crpcut redefines some macros.
There's an important difference when running
google-mock under crpcut
compared to running under
google-test. Unlike
google-test, crpcut always
terminates the test case immediately when a violation is detected.
google-mock interacts very
nicely with crpcut fixtures. Here's an example of letting several test cases use the
same mocks, in different sequences.
#include <gmock/gmock.h>
#include <crpcut.hpp>
struct iface1 {
virtual void f1(int) = 0;
};
struct iface2 {
virtual void f1(const char *) = 0;
};
struct mock1 : iface1 {
MOCK_METHOD1(f1, void(int));
};
struct mock2 : iface2 {
MOCK_METHOD1(f1, void(const char*));
};
struct fix_base {
mock1 m1;
mock2 m2;
};
class fix_seq1 : public fix_base {
public:
fix_seq1() {
EXPECT_CALL(m1, f1(3)).InSequence(s);
EXPECT_CALL(m2, "hello").InSequence(s);
}
private:
testing::sequence s; // prevent test cases from changing the sequence
};
class fix_seq2 : public fix_base {
public:
fix_seq2() {
EXPECT_CALL(m1, f1(3)).InSequence(s1, s2);
EXPECT_CALL(m1, f1(4)).InSequence(s1);
}
protected:
testing::sequence s1; // open-ended, for test-cases
testing::sequence s2; // to add more to the sequences
};
TEST(t1, fix_seq1)
{
... // do things with m1 and m2
}
TEST(t2, fix_seq2)
{
// add more to the started sequence
EXPECT_CALL(m2, f("hello")).InSequence(s2);
... // do things with m1 and m2
}
TEST(t3, fix_seq2)
{
// add other things to the started sequence
EXPECT_CALL(m2, f("world")).InSequence(s2);
... // do things with m1 and m2
}
The main program
The normal main() is exactly the below:
int main(int argc, char *argv[])
{
return crpcut::test_case_factory::run_test(argc, argv);
}
crpcut::test_case_factory::run_test() expects parameters as a set of
flags followed by a set of test case or test suite names. The flags are:
| flag |
|
meaning |
| -c num |
|
Control the number of parallel spawned test
case processes. The default is 1. -c cannot be
combined with -s (below.) |
| -d name |
|
Set working dir for test to named directory. The
directory must exist prior to run, and should
preferably be empty. By default a directory is
created under /tmp/crpcut?????? |
| -l |
|
List, on stdout, all test cases matching the test case or test
suite names, then exit with code 0.
|
| -n |
|
Ignore dependencies when running tests. |
| -o file |
|
Direct xml output to named file. Brief result
will be displayed on stdout. |
| -p name=value |
|
Create a named command line parameter to be
accessed from test cases. |
| -q |
|
Don't display the -o brief |
| -s |
|
Run a single test case, and run it in the main
process. Useful for running a test case in a debugger. -s
implies -n (above) and cannot be combined with -c (also
above.) |
| -v |
|
Include the result of successful tests, in addition to that
of the failed ones, in the test report. |
| -x |
|
Print XML-output on stdout or non-XML to named file
(-o above) |
Note that -s provides an environment that is markedly different
from a normal run in a child process. Most notably no post-mortem checks are
made (files left behind, processing of termination reason.) Also INFO and
FAIL streamers (above) just print to stdout/stderr. Exit code from the run
will be 0 on success, and whatever the process exits with on failure
(typically SIGABRT from the ASSERT_* macros.) Tests that die,
will also terminate the program, even if death is expected.
To run the tests in test suite named "asserts", 8 test cases in
parallel, ignore all dependencies, and print also successful tests, start the
test program using -n -v
-c
8 asserts.
The exact prototype for run_test is:
namespace crpcut {
class test_case_factory
{
public:
static int run_test(int argc, const char *argv[], std::ostream & = std::cerr);
};
}
The return value is the number of failed tests, or -1 if anything was printed
on the stream. The output stream is where to print information if the
parameters in the call don't make sense.
The result from a test run is printed on stdout, or the file named with
the -o flag. By default, the result on stdout is a human readable
format, while output to a named file (the -o flag) is formatted using
the XML Schema provided in crpcut.xsd. The -x flag inverts
the output formats so output on stdout becomes XML-formatted and named file
output becomes human readable.
The test program must be linked with libcrpcut.so. If you use
google-mock in your test
cases, you must also link with libgmock.so and libgtest.so.
Debugging
Debugging a test case is easy. Load the test program into your favourite
debugger. Set a break point on testcasename::test. Run with
-s testcasename. Example:
>$ gdb --args ./test/testprog -s asserts::should_succeed_assert_no_throw
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu"...
(gdb) break asserts::should_succeed_assert_no_throw::test
Breakpoint 1 at 0x487ce4: file /home/bjorn/devel/crpcut/test-src/asserts_and_depends.cpp, line 80.
(gdb) run
Starting program: /var/tmp/build/test/testprog -s -n asserts::should_succeed_assert_no_throw
[Thread debugging using libthread_db enabled]
[New Thread 0x7f5605d43700 (LWP 17769)]
[Switching to Thread 0x7f5605d43700 (LWP 17769)]
Breakpoint 1, asserts::should_succeed_assert_no_throw::test (this=0x7fff0dd76340)
at /home/bjorn/devel/crpcut/test-src/asserts_and_depends.cpp:80
80 ASSERT_NO_THROW(i=1);
(gdb) quit
From there, you can single step the test case.
Advanced techniques
Heap management
Testing how your software behaves in low memory situations can be difficult.
With crpcut you can limit the number of bytes that can be allocated on the
heap. Any attempt to allocate more will fail (malloc() returns 0
and new throws std::bad_alloc.) The limit is expressed
in absolute bytes allocated, using crpcut::heap::set_limit(size_t).
The value crpcut::heap::system can be used to remove all limitations.
To know what limit to set, use the heap inspection function
crpcut::heap::allocated_bytes() and add a suitable number.
TEST(fail_vector_resize)
{
std::vector<int> vec;
crpcut::heap::set_limit(crpcut::heap::allocated_bytes() + 100);
vec.resize(50); // throws!
}
Another heap inspection function is
crpcut::heap::allocated_objects(). The heap inspection
functions can be used to verify that your tests do not leak memory.
TEST(verify_no_leaks)
{
size_t pre_bytes = crpcut::heap::allocated_bytes();
size_t pre_objs = crpcut::heap::allocated_objects();
function_suspect_of_memor_leak();
ASSERT_EQ(pre_objs, crpcut::heap::allocated_objects());
ASSERT_EQ(pre_bytes, crpcut::heap::allocated_bytes());
}
File system access
Since all test cases run in their own working directory, the path to the
working directory when starting the test suite is lost. If you need that
path, for example to populate test vectors from a file, you can get the
path name through the static member function:
const char *crpcut::test_case_factory::get_start_dir();
Named command line parameters
Named command line parameters, added to the test run with the -p
switch, can be accessed from within the test cases using:
class test_case_factory
{
public:
static const char *get_parameter(const char *name);
template <typename T>
static void get_parameter(const char *name, T& t);
template <typename T>
static T get_parameter(const char *name);
};
The first returns the raw value of the parameter, or 0 if none was found.
The other two fails the test case if the value is not found, or cannot be
interpreted as the desired type by means of input streaming.
Predicates
A predicate is anything that looks like a function and returns a type that can
be evaluated as a boolean expression. A predicate can be a free function
pointer, a static member function, or an object with operator().
The BOOST C++ library plays especially
well with predicates, particularly boost::bind and
boost::lambda. Take for example:
const char *p = afunc();
ASSERT_PRED(boost::bind(std::strcmp, _1, _2) == 0, "expected", p);
You can also make a thin wrapper using your own templates:
template <typename R, typename P1, typename P2>
struct binder<R (*)(P1, P2)>
{
typedef R (*proto)(P1, P2);
typedef decltype boost::bind(proto(0), _1, _2) type;
static type bind(proto f) { return boost::bind(f, _1, _2); }
};
template <typename F>
typename binder<F>::type return_of(F f)
{
return binder<F>::bind(f);
}
The above thin wrapper can be used as:
const char *p = afunc();
ASSERT_PRED(return_of(std::strcmp) == 0, "expected", p);
Should the above fail, the violation message will be:
ASSERT_PRED(return_of(std::strcmp) == 0, "expected", p)
param1 = expected
param2 = otherval
If a predicate has an output stream operator, it will be used to add
further information to violation messages.
An alternative to the wrapper above may be:
class equal
{
public:
equal(const char *ref) : r(ref) {}
bool operator()(const char *v) const { return strcasecmp(v, r) == 0; }
friend std::ostream& operator<<(std::ostream &os, const equal &e)
{
return os << "case insensitive string equal to \"" << e.r << "\"";
}
private:
const char *r;
};
Its use can be:
const char *p = afunc();
ASSERT_PRED(equal("expected"), p);
A violation message would be:
ASSERT_PRED(equal("expected", p))
equal("expected") : case insensitive string equal to "expected"
param1 = otherval
Writing your own matchers
A matcher is a value testing class that is instantiated via the
match<> function template. A matcher instance can be
used as a predicate (see above) or tested directly as a boolean value.
By default match<T>(...) returns an instance of T(...).
match<>() is overridden for 0 up to 9 templated parameter
values.
To be used as a predicate, the matcher must implement
bool operator() with a suitable number of parameters.
Beware that since the match<T>() function template will
return a T by value, the matcher T must be copy constructible
(or more accurately, it must be move constructible.)
It is advisable to define an output streaming operator for a matcher.
The output stream operator will only be called when the predicate
has failed, and here extra information can be added that makes it
easier to understand the reason for the failure.
Should instantiating T directly not be suitable, the return value
from the instantiation of match<T>(...) can be controlled
by specializing the crpcut::match_traits<T,...>
template (where ... is the parameters to the match<T>(...)
call. This way the type returned can be controlled also by the types of the
parameters to match<>().
A complete, if somewhat contrived example, testing that two pointers
point to the same value.
class ptr_deref_eq
{
public:
template <typename T>
class type
{
public:
type(T* refp) : rp(refp) {}
type(const type<T>& t) : rp(t.rp), cp(t.cp) {}
bool operator()(T* compp)
{
cp = compp; // store values for use by output stream operator
return *rp == *cp;
}
friend std::ostream& operator<<(std::ostream &os, const type<T>& t)
{
os << "reference ptr=" << t.rp << " pointing to:" << *t.rp
<< "\ncompared ptr=" << t.cp << " pointing to: " << *t.cp;
return os;
}
private:
T* rp;
T* cp;
};
};
namespace crpcut {
template <typename T>
struct match_traits<ptr_deref_eq, T*>
{
typedef typename ptr_deref_eq::template type<T> type;
};
} // namespace crpcut
It's use can be:
int *p1 = afunc();
int n = 3;
ASSERT_PRED(crpcut::match<ptr_deref_eq>(&n), p1);
Wrapping library functions
crpcut accesses all functions it uses from libc and librt
directly from the lib by means of dlsym(). This means that you can
define your own versions of those functions in global namespace, if you need it
for your testing. Be aware, however, that the C++ standard library may
use some of these functions, and so may
google-mock. So,
while creating your own
int write(int fd, const void *addr, size_t bytes)
will not harm crpcut, it will probably cause std::ofstream to
misbehave.
If you need to create wrappers of your own, you can easily do that
through the macros and traits templates available in crpcut.hpp.
CRPCUT_WRAP_FUNC(libname, funcname, rettype, prototype_params, call_params)
| macro param |
|
meaning |
| libname |
|
name of the library in which the function resides.
Predefined names are libc and
librt
|
| funcname |
|
The name of the function, without quotes. |
| rettype |
|
The return type of the function. |
| prototype_params |
|
Types and names for all parameters to the
function, enclosed in parenthesis. |
| call_params |
|
The names from prototype_params without
type information, comma separated and enclosed in
parenthesis. |
For functions without return value, there's also a CRPCUT_WRAP_V_FUNC()
macro. It has the exact same parameter list (including rettype, which
typically is void, but may optionally include some attribute, like
__attribute__((noreturn)).) The original function will be accessible
through the namespace from which CRPCUT_WRAP_FUNC() was instantiated.
This is easiest to explain through an example. Say you want to intercept
all calls to fopen(). You want it to do what it normally does,
but you want to validate its parameters. A way of doing that is as follows:
extern "C" {
#include <stdio.h>
}
#include <crpcut.hpp>
namespace mywraps {
CRPCUT_WRAP_FUNC(libc, fopen, FILE*, (const char *p, const char *m), (p, m))
}
FILE* fopen(const char *p, const char *m)
{
ASSERT_TRUE(p);
ASSERT_TRUE(m);
return mywraps::fopen(p, m);
}
By default the libraries libc and librt are known by
crpcut. If you want to access functions in other libraries, you add your
own constants under the namespace crpcut::libs. These must be
positive integers. You also provide your own specialization of the
traits template
const char *crpcut::libwrapper::traits<int>::name[].
The last element in the name array must be 0. crpcut tries the
library names in the order they appear in the name array, until it
succeeds. If no library can be found, crpcut will segfault.
A complete example for the function asin() in the libm
library:
extern "C" {
#include <math.h>
}
#include <crpcut.hpp>
namespace crpcut {
namespace libs {
static const int libm = 1;
}
namespace libwrapper {
template <>
const char *traits<libs::libm>::name[] = {
"libm.so",
"libm.so.6",
0
};
}
}
namespace mymath {
CRPCUT_WRAP_FUNC(libm, asin, double, (double d), (d))
}
Now the function mymath::asin() will call the original
function in libm, and you can define your own asin() in
global namespace for testing purposes.
extern "C"
{
double asin(double d)
{
ASSERT_GE(d, -1.0);
ASSERT_LE(d, 1.0);
return mymath::asin(d);
}
}
Further development
The todo list, without any regard to importance or desired implementation
order is:
- Rewrite documenation before 1.0
- Set regexp match rules for stdout and stderr (probably won't happen before 1.0)
- OS/X compatibility would be nice