Sign in to follow this  

Adding suites to a unit testing framework

This topic is 3595 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Adding suites to a unit testing framework I've got a custom unit testing framework I've been tinkering with for a while, based on CppUnitLite. It's mostly focused on being really easy to add and run tests, so it looks something like this:
// .cpp file only, no .h needed!
TEST(YourTestName)
{
  // test code here
}

And to run:
For all tests
AllTests.exe *

For a subset
AllTests.exe Renderer*

Now I'm getting quite a few tests, it takes a while to run them all (just under a minute). It'd be nice to be able to run a subset while writing new code (eg. just run the rendering tests since I'm unlikely to break the sound tests while tinkering with render code). The obvious solution would be to define test suites which can be used to group tests and then run a relevent subset. Idea 1 Don't allow nested suites (ie. suites can't contain other suites) and extend the TEST macro to include the test suite name. Something like:
// Code
TEST(YourTestName, Renderer)
{
  // ..
}

// To run all
AllTests.exe *

// To run a suite
AllTests.exe Renderer::*

// To run a test (possibly a bit wordy for my liking, maybe just "YourTestName" and find in all suites?)
AllTests.exe Renderer::YourTestName

It's reasonably simple and consistent. But it might be restrictive not to allow nesting of suites. For example I might want to define a Renderer suite, and within it a Renderer::Geometry suite. However something similar might be achived with a suitable naming convention (eg. Renderer:: and RendererGeometry:: so they can be run with Renderer*::*) Idea 2 Separate out the test definition from the suite definition, allowing for tests to be in multiple suites and suites to also be contained within suites. Something like:
TEST(Test1) {}
TEST(Test2) {}

ADD_TO_SUITE(Test1, Suite1)
ADD_TO_SUITE(Test2, Suite2)
ADD_TO_SUITE(Suite1, Suite2)

This would appear to be much more flexible, and you could pretty much arrange the tests and suites however you want. However it makes running the tests consistantly awkward, since there won't be a single identifying "path" for any given tests (and unlike the first method we can't use the suite names as namespaces for the test names to avoid collisions). Idea 3 Abuse the __FILE__ macro to automatically create test suites based on the directory sturcture of the test files. This is tempting since it's largely automatic and sensible. But I suspect it's a little to "magical" and could have some serious downsides that I havn't spotted yet. Anyone any suggestions for improvements or alternative methods? Cheers

Share this post


Link to post
Share on other sites
After some more thought I've gone off the idea of abusing the __FILE__ macros since it's all a bit too magical and I'm not sure that grouping tests by path is always going to be valid - some people might prefer to keep their tests in the same dir as the original code, for example, while I like to keep them in a seperate sub dir. Plus I think suites should be coarser than directory structures (since I tend to go for lots of little nested directories within each module).

Still not sure on which other the others would be a better idea though. The first seems more logical and consistant, but not allowing suites to be nested inside other suites seems like an arbitrary restriction which doesn't do anyone any good.

Share this post


Link to post
Share on other sites
Another idea is to put suites (of some appropriate granularity) in to dynamic libraries.

Then a configuration file could be used to group/layer the suites appropriately, or the test runner could just scan a directory tree for dynamic libraries that expose a particular interface and use the folder structure as levels in a hierarchy.

I haven't tried this, but I've often thought about it. Currently, I just use Python a script to search for test programs, which is almost the same thing I guess.

Share this post


Link to post
Share on other sites
An interesting idea, but not really what I have in mind for this framework. The main focus is on making writing and building tests dead easy - otherwise no-one does! Which means things like changing the build method to spit out lots of individual libraries and having external configuration files are a big no-no. Plus it still doesn't actually address the problem of how you define suites in the first place.

IMHO scanning for classes implementing a particular interface or naming convention is a technique best kept for languages with good reflection (Java, C#) rather than C++ (and IIRC is how JUnit and NUnit actually do things).

Share this post


Link to post
Share on other sites
Quote:
Original post by OrangyTang
An interesting idea, but not really what I have in mind for this framework. The main focus is on making writing and building tests dead easy - otherwise no-one does! Which means things like changing the build method to spit out lots of individual libraries and having external configuration files are a big no-no.


Well, I'd say that's a problem with your build system. It's a piece of cake for me :)

Quote:
Plus it still doesn't actually address the problem of how you define suites in the first place.


Each dll/so/dylib/whatever defines a suite. Just create an entry-point/export that runs the tests.

Quote:
IMHO scanning for classes implementing a particular interface or naming convention is a technique best kept for languages with good reflection (Java, C#) rather than C++ (and IIRC is how JUnit and NUnit actually do things).


All you'd need to do is LoadLibrary/dlopen/whatever and search for a function called 'run_tests'.

The solution may be much more low-tech than you realise!

Share this post


Link to post
Share on other sites
Quote:
Original post by the_edd
Quote:
Original post by OrangyTang
An interesting idea, but not really what I have in mind for this framework. The main focus is on making writing and building tests dead easy - otherwise no-one does! Which means things like changing the build method to spit out lots of individual libraries and having external configuration files are a big no-no.


Well, I'd say that's a problem with your build system. It's a piece of cake for me :)

Quote:
Plus it still doesn't actually address the problem of how you define suites in the first place.


Each dll/so/dylib/whatever defines a suite. Just create an entry-point/export that runs the tests.

Quote:
IMHO scanning for classes implementing a particular interface or naming convention is a technique best kept for languages with good reflection (Java, C#) rather than C++ (and IIRC is how JUnit and NUnit actually do things).


All you'd need to do is LoadLibrary/dlopen/whatever and search for a function called 'run_tests'.

The solution may be much more low-tech than you realise!

Sure, I'll just restructure the entire team's build system just to add test suite support which could otherwise be done by some small code changes. That's a really productive and practical approach. [rolleyes]

And LoadLibrary or equivilent may not be available on all platforms in a form suitable for actual use.

Share this post


Link to post
Share on other sites
Quote:
Original post by OrangyTang
Sure, I'll just restructure the entire team's build system just to add test suite support which could otherwise be done by some small code changes. That's a really productive and practical approach. [rolleyes]


Well sorry, but nowhere in your original question did you say anything about:
* basic build system restrictions
* this being for a team
* there being a large body of existing work to modify
* platforms without dynamic libraries had to be supported.

By saying you're "still tinkering with it", I could have only assumed that you were still in an early stage of planning and therefore suggestions such as mine were perfectly reasonable!

If your environment can't handle it then fine, but it's actually a very simple system.

Share this post


Link to post
Share on other sites

This topic is 3595 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this