How to handle crashes under undefined circumstances?

Started by
2 comments, last by frob 6 years, 11 months ago

Hi, so I was running complete test of my game. Everything went OK, but then it suddenly crashed after 20 minutes of gameplay, and it was in no special state like loading map or so. It just crashed right during game, and after a long time. I would like to know what is the common approach in this case - when you have no idea where exactly it could crash.

I am not multithreading anything, so debugging will be a bit easier because of it. Should I wrap everything into one big try and catch the exception, in which I will log some data? Or how are done those crash handlers, that provide some info to you?

Advertisement
If it is only after some time and random it could be a memory leak. Try to monitor your ram usage, if it is steadily increasing you have a memory leak.

If it is only after some time and random it could be a memory leak. Try to monitor your ram usage, if it is steadily increasing you have a memory leak.

So far it happened twice and it was always on the same map and always approx after 22 minutes. I will try to monitor RAM, as you say.

But even if it was a memory leak, I would anyway like to know how to print some info about bug, to know on what place it happens. Fprintfs on loop arent helping cause after crash the data stream is left open so the part I am interested in (end of log file) is deleted after that. Printfs on console are technically possible but slow. I am interested in proper way how its done.

Everything went OK, but then it suddenly crashed after 20 minutes of gameplay, and it was in no special state...

How did it crash? Is it something you can crash in the debugger? Does it generate a crash dialog? Exception dialog? Did it just vanish suddenly? Have you built a crash handler into your program, or set the program to generate minidumps?

The first step is to collect information. The way it crashes can make a difference in the collected information. If you can get line numbers and stack traces that is the most useful. Otherwise, the best you can do is collect general information and hope that eventually the solution is found.

There are occasionally bugs like that in long-running programs and engines, bugs that take years for someone to finally discover. Hopefully yours won't be that bad, yet it is something to be aware of.

Printfs on console are technically possible but slow. I am interested in proper way how its done.

There are many different debug heaps and tools for debugging memory issues. For example, Visual Studio has a fairly good debug heap (although it is not always easy to reach all of its features). You can track each allocation by number, you can record stack traces for every allocation which survive in the crash dump. But that only works if you properly generate the stack dump or can capture it in the debugger upon crashing. VMMap is a Microsoft tool to help you view your memory use. If those don't work for you there are other memory management tools and libraries and debug heaps. Googles' TCMalloc libraries, for example, or DMalloc or Valgrind or BoundsChecker.

If you suspect it is an allocation issue, you may want to add more alerts when stack space runs low (perhaps <64KB on a thread) or total heap space drops too loo (perhaps <512 MB) or when the largest allocatable object drops too small (perhaps <256 MB).

Logging every allocation to disk is not likely to help unless most other avenues have been tried, since that will be a serious performance issue and your game needs to run for 20+ minutes before the issue happens. It might help you, but it wouldn't be my first choice of tools.

This topic is closed to new replies.

Advertisement