• Advertisement
Sign in to follow this  

Dark memory voodoo

This topic is 3576 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Anyone know any possible reasons why an app would have over 600MB of reserved but uncommitted virtual memory? Tracking calls to VirtualAlloc shows only about 150MB of actual requests, and the factor of 4 difference between the two figures can't be explained by mere page size rounding. Running a set of calls to HeapWalk indicates that the actually committed memory volume is about where it should be, but the number of reserved pages is huge by comparison. A run of VirtualQuery shows some pretty nasty heap fragmentation, so that's the best bet - but we're not convinced it's the entire issue just yet. We're kind of stumped by this one, so if anyone has some ideas, dump 'em forth.

Share this post


Link to post
Share on other sites
Advertisement
Hmm, very interesting. My guess is that whenever you free memory the OS un-commits the memory like it should, however it keeps the address space reserved for later use. So 600MB of reserved memory represents the maximum amount of memory you've had committed at any one point in the application's lifetime (assuming it wouldn't reserve and commit more memory if it already had enough un-committed reserved memory).

Really though, without debugging the application it's just a guess :) What platform is this BTW?

Share this post


Link to post
Share on other sites
This is on Windows (sorry, thought it'd be obvious from the WinAPI functions).

The reservation theory is a nice one, but it's easily shot down - I wrote a small app that makes several huge random-sized allocations, and then frees them; the result is that the assigned pages are marked Free (vs. marked Reserved) afterwards. So unless the OS has some truly impressive psychic abilities about the app's memory usage patterns, I think we can rule that one out [wink]

Which, of course, leaves us with a total mystery.

Share this post


Link to post
Share on other sites
Quote:
Original post by ApochPiQ
This is on Windows (sorry, thought it'd be obvious from the WinAPI functions).

Well, it could have been an XBox 360 [grin] In which case I might have expected something strange like this :)

Is it possible to use VirtualProtect[Ex] -- or any other mechanism -- to trigger a page fault whenever someone tries to change the state of the page (i.e. from free to reserved)? Because then you might be able to sprinkle a few of these protected pages around in the 450MB of virtual memory that should be free and track when it's being reserved. You could also just protect them against access of any kind, but that wouldn't help if the pages are reserved but never accessed. Or, does changing the state count as an access?

Share this post


Link to post
Share on other sites
I think it's pretty common and I'd believe that heap fragmentation is entirely the reason.

For example, right now VCExpress has 58,176 K Private bytes and 256,088 K Virtual size. (approximately a factor of four)
Other programs are much worse e.g. 1,768 K Private and 38,908 K Virtual.

We're actualy having problems at work with one of our processes reaching about 1.8GB Virtual Size for one customer and the customer is about to commision more devices to the system which might push this to the limit. If we learn anything useful to combat this I'll let you know. But unless you get near 2GB it might not be worth worrying about.

Share this post


Link to post
Share on other sites
All of the memory space is used at one point or another - it's during the shutdown process that things get wonky. According to our code the pages should be getting freed straight up, but for some reason we end up with a huge amount of fragmentation and a lot of reserved but not committed or freed pages.


And yes, we're getting very close to the magic 2GB barrier - in fact, that's what prompted the exploration of the virtual memory situation in the first place.

Share this post


Link to post
Share on other sites
I'm guessing you didn't just post that in six seconds [smile]
Okay, well it sounds like you're having the same issues we are then.

So far I've found a thread leak in our process. Threads that were supposed to be getting destroyed were actually deadlocking with themselves before they could end, and then being forgotten about. I've fixed that.

Another thing is we've decreased the default stack size for threads since our app has tons of threads and probably doesn't ever come close to needing even 128KB stack let alone 1MB. Oh and we're talking an excess of 1500 threads here so it makes a huge difference.

Share this post


Link to post
Share on other sites
We're not running more than a couple of backup threads for auxiliary stuff, so unfortunately that's not a particularly promising avenue.

It's looking more and more like heap fragmentation is the sole cause, so the real trick is going to be finding a way to minimize the fragmentation without rewriting half the code...

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement