I have a server program I made on my linux server that runs for 8 hours and then crashes overnight. I need it to run constantly.
How do I find out what is causing the crash?
It's keeping an ODBC MySQL connection open and checks for UDP packets on port 19023 using recvfrom.
Like the others said your best bet is to get a core dump file for the application. You can then step through the core dumb with gdb to find where your crash is happening. Since it is a 8hr run odds are you have a memory leak somewhere as if it was a null pointer being accessed it would probably crash much much sooner. Read the man pages for core as it will explain how you use it if you are not getting actual core dumps. The general gist of coredumps is you can set a dumpfile and start your daemon and when it crashes it will write the report to that file. Sadly it is really the only way on linux to solve the problem besides invoking the daemon attached to GDB.
There is actually another option besides the above but I don't know how useful it will be for this situation. The command is called strace.
You can run strace -o /path/to/output.txt /path/to/daemon
The command will log everything that application does including the crash. It would give you reason for the crash but not the exact execution point of the crash.
It just crashed with valgrind and it didn't say there were any memory leaks.
It seems it crashes whenever my open SSH session gets closed by me closing the laptop or leaving it on until it goes to sleep.
I thought I had turned on core dump following these instructions
http://www.cyberciti.biz/tips/linux-core-dumps.html
but I don't see anything in /tmp.
So to run it under gdb I type in
gdb myProgram
run
But since I'm doing this remotely from an SSH shell I need it to fork the process and output to a file. How do I do that? I need it to continue running after I end the SSH session.
I can't just do
gdb myProgram &
because I need to type in "run" to gdb.
Oh, that's it. whenever I close the SSH session it crashes.
I thought it was enough to add "&" to fork the process.
How do I get it to continue running after I quit SSH?
Even though you found a solution, you might want to look into either tmux or [gnu]screen. They are basically terminal multiplexers that allow you to keep your session alive when you disconnect and then you can reattach to your previous session again by typing e.g "tmux attach".
Very handy, especially if you develop the software itself on a remote server.
