Sign in to follow this  
polyfrag

Linux server process crashing after ~8 hours

Recommended Posts

polyfrag    2502

I have a server program I made on my linux server that runs for 8 hours and then crashes overnight. I need it to run constantly.

 

How do I find out what is causing the crash?

 

It's keeping an ODBC MySQL connection open and checks for UDP packets on port 19023 using recvfrom.

Share this post


Link to post
Share on other sites
alvaro    21246
What SiCrane said. If it does not leave a core dump, learn how to run it so it does. You can also run your server from gdb, or attach gdb to the running program before it crashes.

Besides that, check if the process is growing in size, which would mean you have a memory leak of some sort.

Share this post


Link to post
Share on other sites
blewisjr    752

Like the others said your best bet is to get a core dump file for the application.  You can then step through the core dumb with gdb to find where your crash is happening.  Since it is a 8hr run odds are you have a memory leak somewhere as if it was a null pointer being accessed it would probably crash much much sooner.  Read the man pages for core as it will explain how you use it if you are not getting actual core dumps.  The general gist of coredumps is you can set a dumpfile and start your daemon and when it crashes it will write the report to that file.  Sadly it is really the only way on linux to solve the problem besides invoking the daemon attached to GDB.

 

There is actually another option besides the above but I don't know how useful it will be for this situation.  The command is called strace.

 

You can run strace -o /path/to/output.txt /path/to/daemon

 

The command will log everything that application does including the crash.  It would give you reason for the crash but not the exact execution point of the crash.

Share this post


Link to post
Share on other sites
kop0113    2453
Since it crashes over time, it does indeed sound like a slow memory leak.

My suggestion is to compile everything with '-g' (in GCC) to add debugging symbols.

Then run it in "valgrind --tool=memcheck" (Which you should really do before the output leaves the development machine anyway).

This should then tell you all allocations that are not subsequently deallocated so you should be able to pinpoint the issue pretty easily. Edited by Karsten_

Share this post


Link to post
Share on other sites
polyfrag    2502

It just crashed with valgrind and it didn't say there were any memory leaks.

 

It seems it crashes whenever my open SSH session gets closed by me closing the laptop or leaving it on until it goes to sleep.

 

I thought I had turned on core dump following these instructions

 

http://www.cyberciti.biz/tips/linux-core-dumps.html

 

but I don't see anything in /tmp.

So to run it under gdb I type in
 

gdb myProgram
run


But since I'm doing this remotely from an SSH shell I need it to fork the process and output to a file. How do I do that? I need it to continue running after I end the SSH session.

I can't just do

 

 

gdb myProgram &


because I need to type in "run" to gdb.

Edited by polyfrag

Share this post


Link to post
Share on other sites
kop0113    2453

Even though you found a solution, you might want to look into either tmux or [gnu]screen. They are basically terminal multiplexers that allow you to keep your session alive when you disconnect and then you can reattach to your previous session again by typing e.g "tmux attach".

Very handy, especially if you develop the software itself on a remote server.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this