A few days ago, I wanted to test some initial systems I made and see how they would perform on a computer that wasn't my dev pc.

So I handed out some EXEs to my friends, first friend everything runs perfectly. Second friend instant crash! I thought maybe it has something to do with graphics drivers, so they tested it on a different computer... Boom crash again! Having no idea why I added more debug and checkpoint statements. After a hour or so I finally had enough statements to pin point the problem, figured out I never set a GLEW boolean to true

Then this got me thinking about debugging. What happens if something like this happens again? I don't want to go back and add a ton of statements every time a problem happens. Normally I would add some global debug variable to switch debugging on/off, but when I think about it that can't be that great of a solution right? I mean, you'ed have a if check for every block of debug statements. That's got to hurt performance to do all those checks, especially if you have a lot

So I'm wondering what do you guys do? Or am I doing it?

It depends on the programming language you use. It is normal to have at least two builds, one debug build and one release build. In C/C++ they differ mainly by defining a pre-processor variable like _DEBUG and switching off some optimizations and leaving debug symbols in the binaries in the debug build, while having no _DEBUG and doing aggressive optimization and stripping debug symbols in the release build. Using the _DEBUG variable with stuff like #if defined _DEBUG ... #endif allows to have pieces of code only in the debug build. If the project gets bigger, you may introduce a finer control than a global _DEBUG. So you can restrict embedded debug support to specific section / sub-systems / whatever.

One thing that is useful in this sense is the assertion checking (in C/C++ the assert(...) function). There is a paradigm called "design by contract", that verbally means "if the state is A when invoking me, then I guarantee that the resulting state is B". To verify this, assertions called "requirements" or "pre-conditions" at the beginning of functions can be checked, and assertions called "ensurements" or "post-conditions" at the end of functions can be checked. Notice that this means not to wait until a problem occurs. It is meant to be part of the programming process, giving you a tool to feel certain that things work as expected.

Of course, all this helps only if you give your friends / testers the debug build, and if the debug will log the problem before it crashes the program.

The overhead of checking your global flag isn't that great, provided you don't do it too often.

But don't make the mistake of doing other expensive processing, even if the debug flag is off. Don't format complicated debug strings, then pass them to a debug function to be ignored. Instead, do the check first, before the formatting.

---

You might also want to consider enabling post-mortem debugging in your app. How to do this depends on the environment, but it's usually possible to collect some kind of crash dump, and ask the users to send this back, which lets you see what went wrong in a production environment that you can't necessarily replicate. A lot of (e.g. PC) games do this.

It's best to collect crash dumps from executables you distribute. You can then load these crash dumps with the PDBs generated _at the same time you built the executable_ into Visual Studio quite easily (meaning that you must keep your PDBs around for any executable - or DLL - that you distribute).

The problem with adding logging is that you very probably didn't add all the logging you need to debug the problem. Then you're having to continuously deliver updated executables to your long-suffering users, who are continously sending you slightly different log outputs in the hope that the problem is fixed.

With a crash dump, you can immediately inspect the back trace and see exactly where the crash occurred, inspect the stack (to a degree, anyway, since it _is_ a Release mode executable) and even some parts of the program's memory. Basically, you can load up the crash right into your debugger just as if the crash had happened on your own machine, with a few limitations.

I definitely want to add crash dumps to my system to make my future testing easier.
With C++ I'm actually really fuzzy when it comes exceptions/crashes. I pretty much have no experience in this area

I understand what crash dumps are and I know that PDB files are generated with my build (my IDE is visual studio 2013).
But I still do not understand to actual use them. Also what if I want to run my application on OSX or Linux, will those PDB files work?
When I think about it I'm not actually sure how to actually generate a crash dump

I mean it can't be as easy as:

 
void main
{
     try
     {
          /*
           Normal game code. Everything. Update Loop, Render loop, Input code, etc
          */
     }
     catch(exception &e)
     {
           /*Generate Crash dump*/
          /* Create a new file and write all the exception things */
     }
}
 

I understand what crash dumps are and I know that PDB files are generated with my build (my IDE is visual studio 2013).
But I still do not understand to actual use them. Also what if I want to run my application on OSX or Linux, will those PDB files work?
When I think about it I'm not actually sure how to actually generate a crash dump

It won't work in OSX or Linux because they don't use Windows executables. (Well, they can if you are using WINE, but it still works the same.)

It is almost as easy as you wrote.

Create your exception handler with the signature: LONG WINAPI MyCrashHandler(EXCEPTION_POINTERS * ExceptionInfo);

You can put whatever popups and useful information you want, but make sure it calls MiniDumpWriteDump(). That function will write your minidump file.

Then in your main function, register your function with ::SetUnhandledExceptionFilter(MyCrashHandler);

That will capture most stuff, but there are a few situations where the system terminates your program without calling it, like fatally corrupting your heap.

That will capture most stuff, but there are a few situations where the system terminates your program without calling it, like fatally corrupting your heap.

I believe I mentioned this in another thread, but we usually set up a separate launcher that starts the game and attaches itself as a debugger so it can catch the exceptions and write the mini dumps. It's a little more work to get up and running, but being able to catch those extra tricky bugs where the game itself is indeed in a bad state, is totally worth it. Plus you prevent someone from attaching their own debugger to the game, which isn't a huge deterrent for a determined hacker but it's low-hanging fruit that comes for free with the crash dump collector :)