Jump to content

  • Log In with Google      Sign In   
  • Create Account


Linux server process crashing after ~8 hours


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
9 replies to this topic

#1 polyfrag   Crossbones+   -  Reputation: 1715

Like
0Likes
Like

Posted 31 December 2012 - 08:20 AM

I have a server program I made on my linux server that runs for 8 hours and then crashes overnight. I need it to run constantly.

 

How do I find out what is causing the crash?

 

It's keeping an ODBC MySQL connection open and checks for UDP packets on port 19023 using recvfrom.



Sponsor:

#2 SiCrane   Moderators   -  Reputation: 9340

Like
1Likes
Like

Posted 31 December 2012 - 08:27 AM

Does the crashed process leave a core dump?

#3 Álvaro   Crossbones+   -  Reputation: 11719

Like
2Likes
Like

Posted 31 December 2012 - 08:33 AM

What SiCrane said. If it does not leave a core dump, learn how to run it so it does. You can also run your server from gdb, or attach gdb to the running program before it crashes.

Besides that, check if the process is growing in size, which would mean you have a memory leak of some sort.

#4 polyfrag   Crossbones+   -  Reputation: 1715

Like
0Likes
Like

Posted 31 December 2012 - 08:41 AM

It's also holding two ofstream open.



#5 blewisjr   Members   -  Reputation: 558

Like
1Likes
Like

Posted 31 December 2012 - 09:42 AM

Like the others said your best bet is to get a core dump file for the application.  You can then step through the core dumb with gdb to find where your crash is happening.  Since it is a 8hr run odds are you have a memory leak somewhere as if it was a null pointer being accessed it would probably crash much much sooner.  Read the man pages for core as it will explain how you use it if you are not getting actual core dumps.  The general gist of coredumps is you can set a dumpfile and start your daemon and when it crashes it will write the report to that file.  Sadly it is really the only way on linux to solve the problem besides invoking the daemon attached to GDB.

 

There is actually another option besides the above but I don't know how useful it will be for this situation.  The command is called strace.

 

You can run strace -o /path/to/output.txt /path/to/daemon

 

The command will log everything that application does including the crash.  It would give you reason for the crash but not the exact execution point of the crash.



#6 Karsten_   Members   -  Reputation: 1372

Like
1Likes
Like

Posted 31 December 2012 - 10:31 AM

Since it crashes over time, it does indeed sound like a slow memory leak.

My suggestion is to compile everything with '-g' (in GCC) to add debugging symbols.

Then run it in "valgrind --tool=memcheck" (Which you should really do before the output leaves the development machine anyway).

This should then tell you all allocations that are not subsequently deallocated so you should be able to pinpoint the issue pretty easily.

Edited by Karsten_, 31 December 2012 - 10:34 AM.

Mutiny - Open-source C++ Unity re-implementation.
Defile of Eden 2 - FreeBSD and OpenBSD binaries of our latest game.


#7 polyfrag   Crossbones+   -  Reputation: 1715

Like
0Likes
Like

Posted 31 December 2012 - 06:03 PM

It just crashed with valgrind and it didn't say there were any memory leaks.

 

It seems it crashes whenever my open SSH session gets closed by me closing the laptop or leaving it on until it goes to sleep.

 

I thought I had turned on core dump following these instructions

 

http://www.cyberciti.biz/tips/linux-core-dumps.html

 

but I don't see anything in /tmp.

So to run it under gdb I type in
 

gdb myProgram
run


But since I'm doing this remotely from an SSH shell I need it to fork the process and output to a file. How do I do that? I need it to continue running after I end the SSH session.

I can't just do

 

 

gdb myProgram &


because I need to type in "run" to gdb.


Edited by polyfrag, 31 December 2012 - 06:19 PM.


#8 polyfrag   Crossbones+   -  Reputation: 1715

Like
0Likes
Like

Posted 31 December 2012 - 06:24 PM

Oh, that's it. whenever I close the SSH session it crashes.

 

I thought it was enough to add "&" to fork the process.

 

How do I get it to continue running after I quit SSH?



#9 polyfrag   Crossbones+   -  Reputation: 1715

Like
0Likes
Like

Posted 31 December 2012 - 06:38 PM

Oh, figured it out

 

http://en.wikipedia.org/wiki/Nohup



#10 Karsten_   Members   -  Reputation: 1372

Like
0Likes
Like

Posted 01 January 2013 - 09:40 AM

Even though you found a solution, you might want to look into either tmux or [gnu]screen. They are basically terminal multiplexers that allow you to keep your session alive when you disconnect and then you can reattach to your previous session again by typing e.g "tmux attach".

Very handy, especially if you develop the software itself on a remote server.


Mutiny - Open-source C++ Unity re-implementation.
Defile of Eden 2 - FreeBSD and OpenBSD binaries of our latest game.





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS