Back to General and Gameplay Programming

C++ testing frameworks

General and Gameplay Programming Programming

Started by MrMark October 21, 2008 02:00 PM

17 comments, last by Sneftel 15 years, 5 months ago

1,258

October 22, 2008 05:14 PM

You do know that manually registering your tests and all that is optional, right? They have a bunch of macros available for automatically registering tests, and even for organizing them in test suites.

My tests look roughly like this:

#include <boost/test/unit_test.hpp>#define BOOST_TEST_MAINBOOST_AUTO_TEST_CASE( foo_test ){ BOOST_CHECK(2+2 == 4);}BOOST_AUTO_TEST_CASE( bar_test){ BOOST_CHECK_NO_THROW(dostuff());}

And if you want to use a fixture in one of your tests:

struct fixture_class {  fixture_class() : some_member(42) {    // perform set-up for the test  }  ~fixture_class() {    // perform any necessary tear-down after the test  }  int some_member;};BOOST_FIXTURE_TEST_CASE( test_name , fixture_class ){  BOOST_CHECK(some_member == 42);}

I agree, it becomes a major pain if you start fiddling with manually registering your tests in test suites and all that. But as long as you stick with the automatically-registered ones, it doesn't require a lot of work to use. (Then again, I haven't used CppUnitLite, so can't really compare).

Not trying to "convert" anyone though, just curious because if it turns out other frameworks are easier still, I might as well use one of those instead. [grin]

MadKeithV

992

October 23, 2008 01:49 AM

Quote:Original post by jpetrie
I've been using UnitTest++ recently.

I'm a UnitTest++ -user too. At my company we evaluated a number of unit test suites and this one had the best combination of features and ease-of-use at that time. Some niggles: the lack of unicode support, as well as the difficulty in selecting specific tests to run.

I also liked CxxTest which uses Python (or Perl) to pre-process the test files. Easy-to-use, but with a lot of setup requirements that made it less suitable to the development environment here.

It's only funny 'till someone gets hurt.And then it's just hilarious.Unless it's you.

OrangyTang

1,298

October 23, 2008 04:17 AM

Quote:Original post by Spoonbender
And if you want to use a fixture in one of your tests:
*** Source Snippet Removed ***

I agree, it becomes a major pain if you start fiddling with manually registering your tests in test suites and all that. But as long as you stick with the automatically-registered ones, it doesn't require a lot of work to use. (Then again, I haven't used CppUnitLite, so can't really compare).

Not trying to "convert" anyone though, just curious because if it turns out other frameworks are easier still, I might as well use one of those instead. [grin]

Despite adding in fixture support to my own version of CppUnitLite, I've changed my mind and think that explicit fixtures like this are a red herring in C++. The old boost method of adding fixtures (mentioned in my second link, presumably before explicit fixtures were added) is IMHO better:

struct MyFixture{  MyFixture()  {    // .. setup code  }  ~MyFixture()  {    // .. tear down code  }};TEST(Foo){  MyFixture fix;  // .. test code ..}

Advantages:
- Setup/teardown called exactly before a test starts, even if you inherit from another fixture or have other objects as members.
- Doesn't require any explicit support from the unit test framework, making it simpler (especially useful if you're using macros to create tests since it reduces the number of creation macros you need).
- Easier to debug and step into if something goes wrong.
- Can easily add multiple fixtures with explicit initialisation ordering.

Explicit fixtures with setup()/teardown() calls make more sense in languages like Java when you don't have auto objects, but for C++ I can't see why you'd do it any other way.

[size="1"][[size="1"]TriangularPixels.com[size="1"]] [[size="1"]Rescue Squad[size="1"]] [[size="1"]Snowman Village[size="1"]] [[size="1"]Growth Spurt[size="1"]]

Spoonbender

1,258

October 23, 2008 07:50 AM

Quote:Original post by OrangyTang
Advantages:
- Setup/teardown called exactly before a test starts, even if you inherit from another fixture or have other objects as members.
- Doesn't require any explicit support from the unit test framework, making it simpler (especially useful if you're using macros to create tests since it reduces the number of creation macros you need).
- Easier to debug and step into if something goes wrong.
- Can easily add multiple fixtures with explicit initialisation ordering.

Explicit fixtures with setup()/teardown() calls make more sense in languages like Java when you don't have auto objects, but for C++ I can't see why you'd do it any other way.

Yeah, I think it's mostly a matter of convenience. The Boost.Test docs mention the same thing, that explicit support for fixtures was added very late, because it just isn't all that important in C++.

The docs mention these issues with it here:

Quote:
* We need to add a fixture declaration statement into each test case manually.
* Objects defined in fixture are references with "<fixture-instance-name>." prefix.
* There is no place to execute a "global" fixture, which performs "global" setup/cleanup procedures before and after testing.

#1 is a fair point. Boost.Test allows you to register a fixture for an entire test suite. Declaring local variables as you did requires you to add it to every test case manually. Not a big deal of course, but it can save a bit of work.
#2 is of course nothing more than convenience, so no big deal either way.
And #3 is another valid point, something that can't easily be done without support from the testing framework.

But yeah, fixtures aren't essential in C++, and any fixture registered with Boost.Test could just be used as a local variable instead. (Or vice versa, the class you just defined above could be registered as a test suite fixture, since Boost.Test just relies on constructor/destructor too)

OrangyTang

1,298

October 23, 2008 08:41 AM

Quote:Original post by Spoonbender
#1 is a fair point. Boost.Test allows you to register a fixture for an entire test suite. Declaring local variables as you did requires you to add it to every test case manually. Not a big deal of course, but it can save a bit of work.
#2 is of course nothing more than convenience, so no big deal either way.
And #3 is another valid point, something that can't easily be done without support from the testing framework.

Number 4 was the deal-breaker for me - CppUnitLite (and most C++ frameworks) use macros to define each test case. Since you can't (in a cross-platform way) overload the macros based on number of arguments you end up with a big mess of macro names, especially when you want to allow multiple fixtures to be used at once:

TEST(Foo) {}TEST_WITH_FIXTURE(Foo2, FooFixture) {}TEST_WTIH_FIXTURE_2(Foo2, FooFixture, BarFixture) {}TEST_WITH_SUITE(Foo3, SuiteName) {}TEST_WITH_FIXTURE_AND_SUITE(Foo4, FooFixture, SuiteName) {}TEST_WITH_FIXTURE_AND_SUITE_2(Foo5, FooFixture, BarFixture, SuiteName) {}// etc. etc.

Like most unit testing issues it's a trade-off between ease of use, flexibility and verbosity. I would much rather take the simpler and more flexible TEST and TEST_WITH_SUITE over the macro mess above even if it comes out a bit more verbose in the end. YMMV.

[size="1"][[size="1"]TriangularPixels.com[size="1"]] [[size="1"]Rescue Squad[size="1"]] [[size="1"]Snowman Village[size="1"]] [[size="1"]Growth Spurt[size="1"]]

Spoonbender

1,258

October 23, 2008 09:30 AM

Quote:Original post by OrangyTang
Number 4 was the deal-breaker for me - CppUnitLite (and most C++ frameworks) use macros to define each test case. Since you can't (in a cross-platform way) overload the macros based on number of arguments you end up with a big mess of macro names, especially when you want to allow multiple fixtures to be used at once:

Yep, I agree with that one. [grin]
Having to keep track of the different BOOST_REALLY_LONG_MACRO_NAME_THATS_SIMILAR_TO_FOUR_OTHER_MACRO_NAMES macros is a pain... [lol]

the_edd

2,109

October 23, 2008 08:00 PM

Quote:Original post by Spoonbender
Quote:Original post by OrangyTang
Number 4 was the deal-breaker for me - CppUnitLite (and most C++ frameworks) use macros to define each test case. Since you can't (in a cross-platform way) overload the macros based on number of arguments you end up with a big mess of macro names, especially when you want to allow multiple fixtures to be used at once:

Yep, I agree with that one. [grin]
Having to keep track of the different BOOST_REALLY_LONG_MACRO_NAME_THATS_SIMILAR_TO_FOUR_OTHER_MACRO_NAMES macros is a pain... [lol]

*Ahem* excuse the flagrant advertising spiel:

test-o-matic uses variadiac macros (it detects whether they're available) to provide this functionality:

TESTFIX("test name", fixture_mixin1, fixture_mixin2, fixture_mixin3){   // ...}

It's not entirely cross platform as already mentioned, but works on at least the following compilers:

* msvc8 and later
* gcc 3 and later
* borland C++ builder 2006 (aka bcc32 5.82)
* digital mars dmc (from some very early version)
* reasonably recent versions sun studio (untested though, I don't have this)

So it's not *that* un-portable.

[Edited by - the_edd on October 23, 2008 8:00:18 PM]

http://www.mr-edd.co.uk
http://bitbucket.org/edd

OrangyTang

1,298

October 24, 2008 03:57 AM

Quote:Original post by the_edd
So it's not *that* un-portable.

Depends what your platforms are. :)

[size="1"][[size="1"]TriangularPixels.com[size="1"]] [[size="1"]Rescue Squad[size="1"]] [[size="1"]Snowman Village[size="1"]] [[size="1"]Growth Spurt[size="1"]]

Sneftel

1,788

October 25, 2008 05:33 PM

Quote:Original post by Spoonbender
You do know that manually registering your tests and all that is optional, right? They have a bunch of macros available for automatically registering tests, and even for organizing them in test suites.

I'll be damned. I cannot imagine how I missed that my first time through. Thanks.

C++ testing frameworks

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

C++ testing frameworks

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines