From the beginning of designing my game engine, I've decided it would work deterministically. This approach is best if coded like that from start, but could still be possible to implement later in the process:
Being the game deterministic, the engine saves and journals all sources of undeterminism (player keystrokes, random number generator seeds, etc) and saves them to a file. The file is a mere KBs big even after an hour of playing. It's size depends on the amount of undeterminism sources you need to track and the framerate you're running at (i.e. simulating at 60hz needs double the space of a 30hz simulation)
When the game is closed or crashes (I use SEH to catch the crash and save the journal) I can later replay the whole session (I can even fast forward, but how fast I can fast forward depends on how powerful my system is, as it must simulate all frames until the crash without rendering waiting for vsync).
This has proven to be an invaluable tool. I've caught so many crashes in no time (specially those hard-to-reproduce ones), also experimented, and even see through the debugger how a variable's value evolves through every frame.
There are a few caveats:
- A few rare bugs may corrupt the journal, but this is extremely obvious and often blatantly obvious.
- If the bug is caused by an untracked source of undeterminism (i.e. an uninitialized variable is the most common one), the replay will be different on every run. However you can still use this to try to understand when is the moment that your simulation starts to diverge and nail down the source of undeterminism. You can also start removing stuff until the replay is simple enough.
- Sometimes the SEH fails to save the journal, and the replay session is lost.
- Changing code to fix one or more bugs may cause your replay session to diverge before the crash, thus it's no longer useful in seeing if the fix actually works.
I can't tell you how to write your journal, as it is specific to each game engine. However, I point you a few starts:
http://gafferongames.com/networking-for-game-programmers/debugging-multiplayer-games/ -> Keeps a journal for debugging networked games, but it's basically the same thing except for single player, and you're not journaling network packets, but rather key strokes.