Program slowing down, and crashing over time (not a memory leak)
Author
I have a multi-threaded program, which starts out running smoothly, but after a couple hours it starts chugging, and eventually crashes (the entire computer locks up). To me this problem screams memory leak, but I have searched through the code, and tracked memory usage through the task manger, and I am all but positive that there is no memory leak. This happens in debug and release, and I am coding in C++ with VS. So my question is, what are other possible culprits of a program acting like this? My current best guess is that it is either somehow related to threads fighting with one another, and eventually locking themselves up in some infinite loop. Or, there is some kind of message queue which is not being cleared, so as time goes on the program would slow down trying to read through the queue, although I am not convinced this would eventually result in the whole computer locking up (since like I said, memory usage remains constant, and I never see a debug message before the system crashes). I have searched for both of these issues, and have had no luck, I will continue to make these assumptions, unless someone has a helpful suggestion. It could also be useful to note that I am remote debugging, and when the system crashes, the computer running VS simply loses the network connection to the other computer. Thanks in advance for your time.
Personally I think if you have a threading issue it wouldn't last that long. It would go bang quite quick!
Sounds very much like a memory leak to me, although I hear what you say.
Can you monitor the CPU load? What's it like early in the program, and at stages later through to the end?
What can you tell us about the 'remote' part of this? Is that perhaps getting in the way somehow?
Sounds very much like a memory leak to me, although I hear what you say.
Can you monitor the CPU load? What's it like early in the program, and at stages later through to the end?
What can you tell us about the 'remote' part of this? Is that perhaps getting in the way somehow?
So at some point the program starts slowing down, but you can still debug it remotely? You should be able to take a profile snapshot when it starts out and compare it to when it slows down to hunt for clues. It will at least tell you where it is slowing down. Also another trick that works for me once in a while is to simply pause the app via Visual Studio. If you pause it 5 times (when it is slow) and it stops at the same place 4 times, you probably have a solid lead.
Author
Yeah, I am shocked that signs are pointing away from a memory leak. The CPU load is high, even early on (when the program is running fine) it seems to hang out around 100% CPU usage. As for the remote part, the actual program is running on a separate computer (which is running embedded windows). To debug I use VS's feature which allows debugging based on TCP/IP. As for this being a problem, the issue also happens which the program is running in release, completely separate of VS. While this program is running it does have to communicate with multiple USB/RS-232/UDP devices. Unfortunately I can not go into extreme detail about how certain parts of our system work, but if you can think of a detail which I may have left out I will do my best to fill in the information.
Author
OrenGL:
"Also another trick that works for me once in a while is to simply pause the app via Visual Studio. If you pause it 5 times (when it is slow) and it stops at the same place 4 times, you probably have a solid lead."
I actually have tried this already, there is one spot that it usually stops on, it is a message pump for windows messages, which is running in its own thread, this seems suspicious, but I am not completely sure where to go from there.
"You should be able to take a profile snapshot when it starts out and compare it to when it slows down to hunt for clues."
I am not sure I know how to do this, but it sounds useful. If you mean that I should run a 3rd party profiling program, I am not currently set up with one, but from what I have seen in the past, I do think one could be useful.
"Also another trick that works for me once in a while is to simply pause the app via Visual Studio. If you pause it 5 times (when it is slow) and it stops at the same place 4 times, you probably have a solid lead."
I actually have tried this already, there is one spot that it usually stops on, it is a message pump for windows messages, which is running in its own thread, this seems suspicious, but I am not completely sure where to go from there.
"You should be able to take a profile snapshot when it starts out and compare it to when it slows down to hunt for clues."
I am not sure I know how to do this, but it sounds useful. If you mean that I should run a 3rd party profiling program, I am not currently set up with one, but from what I have seen in the past, I do think one could be useful.
Quote:Original post by Ultraseamus
Yeah, I am shocked that signs are pointing away from a memory leak. The CPU load is high, even early on (when the program is running fine) it seems to hang out around 100% CPU usage.
That sounds like a thread is either not idling properly, is running free, or is very busy.
Is it possible the thread is eating memory and not being caught by VS?
I get things like that on both XCode and VS.
Quote:Original post by UltraseamusAs for the remote part, the actual program is running on a separate computer (which is running embedded windows). To debug I use VS's feature which allows debugging based on TCP/IP. As for this being a problem, the issue also happens which the program is running in release, completely separate of VS. While this program is running it does have to communicate with multiple USB/RS-232/UDP devices. Unfortunately I can not go into extreme detail about how certain parts of our system work, but if you can think of a detail which I may have left out I will do my best to fill in the information.
Hmm. Let's not get into that now. Hopefully it's something a little less difficult to track down. :)
A profile is a good idea... You need a WTF setup so that you can wait until the problem gets easier to see, and then kick off the recording...
If you can make your thread delay (passive rather than active) for 10 ms or so every time it pumps events that may resolve issues of the thread hogging resources. It's possible it's just running as fast as it can without giving your processor a break. Your active program loop in almost any application should usually have such a delay depending on what exactly you're doing to do this.
Maybe someone has a better suggestion, but this is what I do to stop my apps from eating 100% cpu when they don't need it. Keep in mind there -is- a performance hit to doing this as you basically have to revert control to your OS for a bit and then regain it on every sleep. Even just sleeping once every x iterations may help.
Maybe someone has a better suggestion, but this is what I do to stop my apps from eating 100% cpu when they don't need it. Keep in mind there -is- a performance hit to doing this as you basically have to revert control to your OS for a bit and then regain it on every sleep. Even just sleeping once every x iterations may help.
Author
Both suggestions seem to be related. It is possible that a thread is sucking up too many resources, but I am not sure if it can be sneaking memory past me, since I am watching the memory usage with windows task manger.
One thing that puzzles me is that most parts of this program ran fine in the past (I think this would include the message pumps) but trying to figure out what has changed, and even when it started breaking is tough (having to wait 1-2 hours for it to slow down is a pain).
I will probably see if I can throw some sleep time into various thread loops, but I am still confused as to how the problem can grow like this over time without it being a memory issue. I would think that if a thread were hogging resources, it would always be hogging those resources.
One thing that puzzles me is that most parts of this program ran fine in the past (I think this would include the message pumps) but trying to figure out what has changed, and even when it started breaking is tough (having to wait 1-2 hours for it to slow down is a pain).
I will probably see if I can throw some sleep time into various thread loops, but I am still confused as to how the problem can grow like this over time without it being a memory issue. I would think that if a thread were hogging resources, it would always be hogging those resources.
Quote:Original post by UltraseamusAlternatives may be tricky in that setup, but the Window's Task Manager is very inaccurate as a memory monitor.
I am not sure if it can be sneaking memory past me, since I am watching the memory usage with windows task manger.
Quote:Original post by Ultraseamus
Both suggestions seem to be related. It is possible that a thread is sucking up too many resources, but I am not sure if it can be sneaking memory past me, since I am watching the memory usage with windows task manger.
That doesn't fill me with confidence, especially with a threaded app. ;)
Quote:I will probably see if I can throw some sleep time into various thread loops, but I am still confused as to how the problem can grow like this over time without it being a memory issue. I would think that if a thread were hogging resources, it would always be hogging those resources.
A bit out there but..
A thread, or the main thread, could be spawning more threads, and in the end the overhead is what's pulling things down.. or some other such madness....
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
