Okay, so I'm writing a .Net application for a research project, and I've run into the problem of memory consumption. The program runs along at a nice steady pace, consuming just over 40 megs of ram, which is about what I expect for the application. However, after a few minutes running, the memory usage suddenly and for no apparent reason starts to climb extremely rapidly and doesn't stop until the program crashes, complaining that it is out of memory. I tried breaking out the good ol' CLR profiler to see what's up, but my program is extremely CPU-intensive, and using the CLR profiler slows it down to a crawl, meaning that it will probably take many hours if not days to run, which makes using the CLR profiler extremely impracticle. So does anyone know of a way to examine what's going on in my app's memory without slowing it down to a crawl? (I tried turning off all the profiling features but to no avail) All I want is a snapshot of what it looks like just before it crashes. Thanks for any tips! - Fuzz

Quote:I just tried the .NET 2 beta 2 CLR profiler and it is much better! You can see a timeline of allocations even! That would be useful for you - if you don't mind running your app for a day...

Yeah, that's the one I have. It does exactly what I need it to do, only too slow.

Quote:Do you make a lot of allocations in your program? If not, I don't see why only profiling allocations should slow it down considerably.

No, I don't really make very many allocations (though clearly I do when the mem usage jumps to 1 gb). I tried turning off both memory profiling and call profiling, but it still ran extremely slowly. I really only need to see the heap dump of the app when the memory has skyrocketed.

Quote:My program, a 3D game is extremely CPU intensive too but is hardly slowed at all by profiling calls and allocations at the same time.
However, I suppose it might be pixel shader bound rather than limited by the CPU but task manager says 100% so I guess that wouldn't make much difference.

Though my app *IS* an MDX program, but it is very much CPU bound since it performs a lot of physics and AI.

I'll recheck my profiling settings. The fact that it's over 100 times slower with all profiling options disabled sounds strange to me.

Hi mate, Im working in research too, using Native C++, Managed C#, GDI+ and DirectX to do a bit of calculations, display some data, blah blah..

Anyway, I had exactly the same problem as you - a piece of software with some pretty tight loops and not excessive memory allocation did its job, then halfway through, memory usage jumped to max (1G + 1G virtual), and the computer crashed. Anyway, I found the problem and sorted it now - its .NET's rubbish garbage collector.

Basically, in 'office' style apps, the pace of CPU usage is too slow to cause the garbage collector to lock up. In a CPU intensive app, the GC cant do its job, as it relies on windows messaging to clean up in idle time.

The solution? DoEvents. Yes, a well placed Application.DoEvents() inside a tight loop allows the GC to do its dirty work.

To get more control over your deallocation, apply the following principles to your code wherever you have problems:

eg:

void myMegaCalculation()
{
// Start horrendous calculation
double [] megaBuffer = new double[hugeNumber];

for (i = 0, j = 0; i < largenumber; i++, j++))
{
megaCalculation1();
megaCalculation2();
displayRoutine();

// Might be necessary, might not. try it and see
if (j == 100)
{
Application.DoEvents();
j = 0;
}

}

// End calculation and clean up
GC.Collect(); // Force the Garbage Collector to get out of bed and collect
Application.DoEvents(); // Give GC some windows down-time to do its job

}


...

Please note, too much GC.Collect() and DoEvents will make your program mega-slow. Like I said. A couple of well placed collections & doevents will do fine. Too many, and your program crawls.

Also check out a program called ANTS Profiler (google it). It profiles memory and execution time of all .NET code. Awesome product, 14 day trial, or $300 for full version I think. Using this helped me to find out it wasn't my program that wasn't releasing memory, but some internal Windows routines that were getting clogged up by so many calcs.

Good luck!!

Andy B

-------------------------------------
Edit, with regards to 3D game programming, in .NET you are forced to use the old Render(); DoEvents loop to allow Garbage Collecting. ie:

void Main (void)
{
// Create a new app
using (myApp = new myCustomApplication())
{
myApp.Show();

// Super tight loop
while(myApp.Created)
{
// all the business happens here
myApp.Render();
// allow some time for Windows to keep up
Application.DoEvents();
}
}
}

Also try threading to split components up into various sections. ie: rendering thread, physics thread, main thread etc... The problem most likely lies with the high CPU stalling the GC. Try ways to reduce or share the CPU loading through some heavy optimisation and well-placed message handling.

You could try Memprofiler (http://memprofiler.com). I've used it in my work in the past and it has worked quite well.

@Toolmaker: I'll try that out... Hmm, it doesn't even crash on my slow laptop here at work. It just sits thrashing the harddrive constantly. The mem usage does jump to 150 mb or so, but then settles back down to around 40. I'll try it out on my fast desktop when I get home.

@Andy: What you say sounds strange to me, from my knowledge of the CLR. It's my understanding that the GC will be invoked when an app runs out of memory given to it by the OS, meaning that allocations of new objects can trigger the GC, in which case you shouldn't have to call DoEvents to make the GC do its job. Besides, then how does the GC work in non-windowed apps? I do call GC.Collect() at reasonably regular intervals (my app runs a lot of independant simulations, and I call Collect between each one since, in this case, I know better than the GC when it is a good time to do a large collection).

However, the evidence of my problem certainly fits what you are describing. My app is designed to perform logic updates at a certain rate and to render/DoEvents as fast as it can when not performing updates. Sometimes it takes too long to perform updates, and the application gets behind where it thinks its supposed to be and stops rendering or calling DoEvents while it trys to catch up. During this period is when the mem usage appears to go crazy.

@AP: thanks, i'll check that out.

Alright, problem solved guys! It turns out that it has nothing to do with DoEvents or the GC not running properly. It was simply a bug in my code that only showed up when the CPU becomes overloaded that caused resource-allocation heavy code to be called in effectively in infinite loop.

So case closed! Thanks all.

- Fuzz