Sign in to follow this  
MrMark

C++ testing frameworks

Recommended Posts

What are some good C++ testing frameworks ? My biggest problem in my last major C++ project was finding exactly where bugs were coming from, and not breaking old code when I introduce new features. This time around I want to be a lot more methodical and a proper testing framework is a good start I think.

Share this post


Link to post
Share on other sites
Obligitory link 1 and two.

Personally I took CppUnitLite and heavily modified it so it was suitable for my purposes. It's working out really well since I get to tune it exactly for my needs, but if you've not got the time then check out the above links and see which features you find most important.

Share this post


Link to post
Share on other sites
Google has one. I think it's fairly new. I've never used it but it looks pretty nice.

On my current project I just rolled my own. I don't think it's that crazy to roll your own, since it doesn't take that much code, and you might want to adapt your testing strategy to the needs of your project.

Share this post


Link to post
Share on other sites
Quote:
Original post by OrangyTang
Personally I took CppUnitLite and heavily modified it so it was suitable for my purposes. It's working out really well since I get to tune it exactly for my needs, but if you've not got the time then check out the above links and see which features you find most important.
Same here, though I found the modifications needed to be pretty minor. CppUnit is nice but a little heavy to apply, and Boost.Test... my goodness. I'm pretty sure unit tests aren't meant to be that painful.

Share this post


Link to post
Share on other sites
Quote:
Original post by SneftelBoost.Test... my goodness. I'm pretty sure unit tests aren't meant to be that painful.


Heh, how do you mean? I haven't yet used it on any large projects, but so far, it's been surprisingly easy to use. Am I just missing something major here? [grin]

Share this post


Link to post
Share on other sites
Quote:
Original post by Spoonbender
Heh, how do you mean? I haven't yet used it on any large projects, but so far, it's been surprisingly easy to use. Am I just missing something major here? [grin]
Perhaps it's just a matter of personal preference. At a minimum, I feel a testing framework should free you from having to do accounting for your own tests. I'd like to write a class, and write tests for it inline, easily and without any repetition or extra boilerplate or files. That's the only difficult part of writing such a framework, after all... the rest pretty much comes down to a big AND clause. Boost.Test makes me put each unit test in a separate class, written out in longhand, manually added to a test suite. I consider that to be too much work. Surely the organization that came up with Boost.Spirit and Boost.Lambda can use a few Stupid C++ Tricks to make that stuff more streamlined?

Share this post


Link to post
Share on other sites
You do know that manually registering your tests and all that is optional, right? They have a bunch of macros available for automatically registering tests, and even for organizing them in test suites.

My tests look roughly like this:


#include <boost/test/unit_test.hpp>

#define BOOST_TEST_MAIN

BOOST_AUTO_TEST_CASE( foo_test )
{
BOOST_CHECK(2+2 == 4);
}

BOOST_AUTO_TEST_CASE( bar_test)
{
BOOST_CHECK_NO_THROW(dostuff());
}




And if you want to use a fixture in one of your tests:

struct fixture_class {
fixture_class() : some_member(42) {
// perform set-up for the test
}
~fixture_class() {
// perform any necessary tear-down after the test
}

int some_member;
};
BOOST_FIXTURE_TEST_CASE( test_name , fixture_class )
{
BOOST_CHECK(some_member == 42);
}



I agree, it becomes a major pain if you start fiddling with manually registering your tests in test suites and all that. But as long as you stick with the automatically-registered ones, it doesn't require a lot of work to use. (Then again, I haven't used CppUnitLite, so can't really compare).

Not trying to "convert" anyone though, just curious because if it turns out other frameworks are easier still, I might as well use one of those instead. [grin]

Share this post


Link to post
Share on other sites
Quote:
Original post by jpetrie
I've been using UnitTest++ recently.


I'm a UnitTest++ -user too. At my company we evaluated a number of unit test suites and this one had the best combination of features and ease-of-use at that time. Some niggles: the lack of unicode support, as well as the difficulty in selecting specific tests to run.

I also liked CxxTest which uses Python (or Perl) to pre-process the test files. Easy-to-use, but with a lot of setup requirements that made it less suitable to the development environment here.

Share this post


Link to post
Share on other sites
Quote:
Original post by Spoonbender
And if you want to use a fixture in one of your tests:
*** Source Snippet Removed ***

I agree, it becomes a major pain if you start fiddling with manually registering your tests in test suites and all that. But as long as you stick with the automatically-registered ones, it doesn't require a lot of work to use. (Then again, I haven't used CppUnitLite, so can't really compare).

Not trying to "convert" anyone though, just curious because if it turns out other frameworks are easier still, I might as well use one of those instead. [grin]

Despite adding in fixture support to my own version of CppUnitLite, I've changed my mind and think that explicit fixtures like this are a red herring in C++. The old boost method of adding fixtures (mentioned in my second link, presumably before explicit fixtures were added) is IMHO better:


struct MyFixture
{
MyFixture()
{
// .. setup code
}

~MyFixture()
{
// .. tear down code
}
};

TEST(Foo)
{
MyFixture fix;

// .. test code ..
}

Advantages:
- Setup/teardown called exactly before a test starts, even if you inherit from another fixture or have other objects as members.
- Doesn't require any explicit support from the unit test framework, making it simpler (especially useful if you're using macros to create tests since it reduces the number of creation macros you need).
- Easier to debug and step into if something goes wrong.
- Can easily add multiple fixtures with explicit initialisation ordering.

Explicit fixtures with setup()/teardown() calls make more sense in languages like Java when you don't have auto objects, but for C++ I can't see why you'd do it any other way.

Share this post


Link to post
Share on other sites
Quote:
Original post by OrangyTang
Advantages:
- Setup/teardown called exactly before a test starts, even if you inherit from another fixture or have other objects as members.
- Doesn't require any explicit support from the unit test framework, making it simpler (especially useful if you're using macros to create tests since it reduces the number of creation macros you need).
- Easier to debug and step into if something goes wrong.
- Can easily add multiple fixtures with explicit initialisation ordering.

Explicit fixtures with setup()/teardown() calls make more sense in languages like Java when you don't have auto objects, but for C++ I can't see why you'd do it any other way.

Yeah, I think it's mostly a matter of convenience. The Boost.Test docs mention the same thing, that explicit support for fixtures was added very late, because it just isn't all that important in C++.

The docs mention these issues with it here:
Quote:

* We need to add a fixture declaration statement into each test case manually.
* Objects defined in fixture are references with "<fixture-instance-name>." prefix.
* There is no place to execute a "global" fixture, which performs "global" setup/cleanup procedures before and after testing.

#1 is a fair point. Boost.Test allows you to register a fixture for an entire test suite. Declaring local variables as you did requires you to add it to every test case manually. Not a big deal of course, but it can save a bit of work.
#2 is of course nothing more than convenience, so no big deal either way.
And #3 is another valid point, something that can't easily be done without support from the testing framework.

But yeah, fixtures aren't essential in C++, and any fixture registered with Boost.Test could just be used as a local variable instead. (Or vice versa, the class you just defined above could be registered as a test suite fixture, since Boost.Test just relies on constructor/destructor too)

Share this post


Link to post
Share on other sites
Quote:
Original post by Spoonbender
#1 is a fair point. Boost.Test allows you to register a fixture for an entire test suite. Declaring local variables as you did requires you to add it to every test case manually. Not a big deal of course, but it can save a bit of work.
#2 is of course nothing more than convenience, so no big deal either way.
And #3 is another valid point, something that can't easily be done without support from the testing framework.

Number 4 was the deal-breaker for me - CppUnitLite (and most C++ frameworks) use macros to define each test case. Since you can't (in a cross-platform way) overload the macros based on number of arguments you end up with a big mess of macro names, especially when you want to allow multiple fixtures to be used at once:


TEST(Foo) {}
TEST_WITH_FIXTURE(Foo2, FooFixture) {}
TEST_WTIH_FIXTURE_2(Foo2, FooFixture, BarFixture) {}
TEST_WITH_SUITE(Foo3, SuiteName) {}
TEST_WITH_FIXTURE_AND_SUITE(Foo4, FooFixture, SuiteName) {}
TEST_WITH_FIXTURE_AND_SUITE_2(Foo5, FooFixture, BarFixture, SuiteName) {}
// etc. etc.




Like most unit testing issues it's a trade-off between ease of use, flexibility and verbosity. I would much rather take the simpler and more flexible TEST and TEST_WITH_SUITE over the macro mess above even if it comes out a bit more verbose in the end. YMMV.

Share this post


Link to post
Share on other sites
Quote:
Original post by OrangyTang
Number 4 was the deal-breaker for me - CppUnitLite (and most C++ frameworks) use macros to define each test case. Since you can't (in a cross-platform way) overload the macros based on number of arguments you end up with a big mess of macro names, especially when you want to allow multiple fixtures to be used at once:

Yep, I agree with that one. [grin]
Having to keep track of the different BOOST_REALLY_LONG_MACRO_NAME_THATS_SIMILAR_TO_FOUR_OTHER_MACRO_NAMES macros is a pain... [lol]

Share this post


Link to post
Share on other sites
Quote:
Original post by Spoonbender
Quote:
Original post by OrangyTang
Number 4 was the deal-breaker for me - CppUnitLite (and most C++ frameworks) use macros to define each test case. Since you can't (in a cross-platform way) overload the macros based on number of arguments you end up with a big mess of macro names, especially when you want to allow multiple fixtures to be used at once:

Yep, I agree with that one. [grin]
Having to keep track of the different BOOST_REALLY_LONG_MACRO_NAME_THATS_SIMILAR_TO_FOUR_OTHER_MACRO_NAMES macros is a pain... [lol]


*Ahem* excuse the flagrant advertising spiel:

test-o-matic uses variadiac macros (it detects whether they're available) to provide this functionality:


TESTFIX("test name", fixture_mixin1, fixture_mixin2, fixture_mixin3)
{
// ...
}





It's not entirely cross platform as already mentioned, but works on at least the following compilers:

* msvc8 and later
* gcc 3 and later
* borland C++ builder 2006 (aka bcc32 5.82)
* digital mars dmc (from some very early version)
* reasonably recent versions sun studio (untested though, I don't have this)

So it's not *that* un-portable.

[Edited by - the_edd on October 23, 2008 8:00:18 PM]

Share this post


Link to post
Share on other sites
Quote:
Original post by Spoonbender
You do know that manually registering your tests and all that is optional, right? They have a bunch of macros available for automatically registering tests, and even for organizing them in test suites.

I'll be damned. I cannot imagine how I missed that my first time through. Thanks.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this