Sign in to follow this  
wqking

Unit test, compile in which build mode -- Debug, Release, Or both?

Recommended Posts

This question can be general and language independent.
But if talking with only C++ is better for you, just let's discuss C++ unit test because I now only unit testing C++ code.

In general we can compile and run unit test in three mode:
1, Compile and run in only Debug mode.
2, Compile and run in only Release mode.
3, Compile in both modes and run twice.

Apparently 3 is the most reliable. But then unit test is a lot more time consuming to compile and run twice.
IMO unit test should be fast.

However, sticking to only one of debug or release mode can't give me 100% confidence because there are usually difference between debug and release mode, especially in C++.

2 is the most closed to the final product, but as it's unit test, running in release mode will miss some important assertions. This is I'm most worrying about.

So now I run my test in debug mode, but the test code behaviors may differ with product.

Your opinion or your experience on which mode you use will be much appreciated.

Thanks

Share this post


Link to post
Share on other sites
If the lack of assertions in release mode bothers you, then you can create a new project configuration with the release mode settings that enable assertions.

Share this post


Link to post
Share on other sites
I keep a second set of debug flags for each subsystem. For example, my memory manager is in debug mode if and only if LSA_DEBUG is set. And it can be set in any configuration, so the rest of the engine can be running in release mode and only the memory manager is performing debug checks.

This is optimal because running the whole project in release mode except for one system is a good way to isolate differences between release mode and debug mode for just that one system.
It prevents the “it only happens in debug mode!!” conundrum from a completely unrelated sub system from impacting your debugging.


L. Spiro

Share this post


Link to post
Share on other sites
At home, I have two different assert macros. "ASSERT" is your typical debug-only assert. "TEST" is used by unit tests, and runs in both debug and release modes.
I run the unit tests for whichever config I'm currently working in at the time (debug or release).

At [i]every [/i]workplace I've had, there have been three build configurations though -- debug, release and retail. Debug and release both have assertions enabled, while retail doesn't (and also doesn't have unit tests).

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1318928819' post='4873820']
At [i]every [/i]workplace I've had, there have been three build configurations though -- debug, release and retail. Debug and release both have assertions enabled, while retail doesn't (and also doesn't have unit tests).
[/quote]

You mean at your work both your debug and release build contains unit test?
As far as I understand, unit test should be a standalone project (that's what I do). So the application doesn't contain any unit test code. Only the test project contains.

Isn't it common method to split app and unit test to two projects?

I didn't have a lot of experience on unit test so hope I won't go wrong from the beginning.

Share this post


Link to post
Share on other sites
At home, I have a solution file which contains many different projects for engine code, game code and unit tests. The unit test projects are set up to run whenever the solution is compiled. If any of them crash or have assertion-failures, it shows up in the "build errors" log of visual studio ([i]with the file/line number of the crash/assert[/i]).

At the places I've worked, they've used both external unit tests, and ones built into the game. Generally the engine tests are an external program, but gameplay tests have to be built into the game.
For example, if you're making a unit test that simulates a full multi-player match to see if any de-sync bugs occur, then you've got to be running the actual game in order to run that test -- so these kinds of tests are compiled into the game ([i]for debug/release builds only, [/i][i]but not included in the [b]retail [/b]build[/i]). With games, it's also a good idea to create unit-tests for performance ([i]fail if frames-per-second too low[/i]) and visual regression bugs ([i]save screenshots to disk as a log of results[/i]), which are also built into debug/release builds.

N.B., the 'debug' build is only used to diagnose programming errors, the 'release' build is used by the game developers ([i]mostly optimised, but still contains asserts[/i]), the retail build is given to customers ([i]fully optimised[/i]).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this