Adding suites to a unit testing framework
I've got a custom unit testing framework I've been tinkering with for a while, based on CppUnitLite. It's mostly focused on being really easy to add and run tests, so it looks something like this:
// .cpp file only, no .h needed!
TEST(YourTestName)
{
// test code here
}
And to run:
For all tests
AllTests.exe *
For a subset
AllTests.exe Renderer*
Now I'm getting quite a few tests, it takes a while to run them all (just under a minute). It'd be nice to be able to run a subset while writing new code (eg. just run the rendering tests since I'm unlikely to break the sound tests while tinkering with render code). The obvious solution would be to define test suites which can be used to group tests and then run a relevent subset.
Idea 1
Don't allow nested suites (ie. suites can't contain other suites) and extend the TEST macro to include the test suite name. Something like:
// Code
TEST(YourTestName, Renderer)
{
// ..
}
// To run all
AllTests.exe *
// To run a suite
AllTests.exe Renderer::*
// To run a test (possibly a bit wordy for my liking, maybe just "YourTestName" and find in all suites?)
AllTests.exe Renderer::YourTestName
It's reasonably simple and consistent. But it might be restrictive not to allow nesting of suites. For example I might want to define a Renderer suite, and within it a Renderer::Geometry suite. However something similar might be achived with a suitable naming convention (eg. Renderer:: and RendererGeometry:: so they can be run with Renderer*::*)
Idea 2
Separate out the test definition from the suite definition, allowing for tests to be in multiple suites and suites to also be contained within suites. Something like:
TEST(Test1) {}
TEST(Test2) {}
ADD_TO_SUITE(Test1, Suite1)
ADD_TO_SUITE(Test2, Suite2)
ADD_TO_SUITE(Suite1, Suite2)
This would appear to be much more flexible, and you could pretty much arrange the tests and suites however you want. However it makes running the tests consistantly awkward, since there won't be a single identifying "path" for any given tests (and unlike the first method we can't use the suite names as namespaces for the test names to avoid collisions).
Idea 3
Abuse the __FILE__ macro to automatically create test suites based on the directory sturcture of the test files. This is tempting since it's largely automatic and sensible. But I suspect it's a little to "magical" and could have some serious downsides that I havn't spotted yet.
Anyone any suggestions for improvements or alternative methods? Cheers
