Jump to content
  • Advertisement
Sign in to follow this  
DvDmanDT

My tool allocates more memory than Windows has available, how to warn user?

This topic is 651 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi,

 

I have a tool that depending on input may need huge amounts of memory. The problem is that the system may run out of memory. The default configuration of Windows starts swapping at this point, and when that's not enough, the default configuration causes a dynamic resize of the swap file, which effectively freezes the entire system. On 32 bit Windows, the address space eventually runs out and allocations fail, causing my app to receive a System.OutOfMemoryException, but that may not happen on 64-bit Windows. At least not in acceptable time. The result is that my app can cause the system to freeze for an unknown amount of time where the only out is a hard reset. I personally think this is a bad and dangerous behavior of Windows, at least for a default, but it is what it is so I need a workaround. It's not realistic to reduce memory usage or switch to some file based solution, manual or memory mapped files.

 

What I want to do at this point is to warn the user when the app is about to use large amounts of memory and ask for confirmation before continuing. To do this, I need to figure out how much memory is "safe" to use at the moment. If the user has 4GB is his system, then perhaps I should only use 1.5GB before warning. If the user has 32GB, I could likely use 20+ without issues. I should probably have some level of respect for other applications running.

 

How would you do this? What system and application properties should I be looking at? Working set, virtual memory usage, commit size, etc?

Share this post


Link to post
Share on other sites
Advertisement

You could write parts of the memory you are using to a file, I doubt you need 2 gb of memory all the time. This is slower tho. Guess you could also write some kind of compresser for memory.

Share this post


Link to post
Share on other sites

Choose some reasonable default, but give the user some command-line switches to tune their memory use -- they might be OK starving other apps of memory more than your default, or might need tighter defaults.

 

Its really not possible to chunk the data up? Perform the operation in passes? Is there any data that goes cold for a time? -- it could be compressed, even if you're not willing to write it out to a file (compression can cost, but decompression is often fast enough that it can beat out uncompressed reads because of bandwidth limitations).

Share this post


Link to post
Share on other sites
I would definitely suggest analyzing how and why you're using that much memory before thinking about adding special cases for limited RAM. Adding special cases will just make your app harder to maintain and potentially add even more things that can go wrong.

Share this post


Link to post
Share on other sites

File based operation is not realistic at all right now. Splitting the input into smaller chunks will be possible at some point in the future, but not soon enough, and doing that mat degrade the usefulness of the tool. Imagine having a giant map and the tool was meant to show pathfinding data, but had to load only parts of the navmesh at a time. There are lots of paths you could find/see, but there are also some paths that couldn't be found/displayed. The results would still be useful but incomplete, and sometimes those missing pieces are what matters.

 

What we need is a quick and hopefully temporary way to not crash the users system when that system can't handle the input, without limiting operation on systems that can handle it. 

Share this post


Link to post
Share on other sites
There might be something buried in the Windows Management Instrumentation system that you can access from C# that gives you enough information to make that kind of decision. I haven't used it very frequently, nor for memory, but I suspect what you're looking for will be in there if it's anywhere.

https://en.wikipedia.org/wiki/Windows_Management_Instrumentation

(edit: I typed "Media" instead of "Management", sorry)


My limited experience using it is that it's SUPER complicated to use, but there's an incredible amount of data you can get from it. Edited by Nypyren

Share this post


Link to post
Share on other sites

Imagine having a giant map and the tool was meant to show pathfinding data, but had to load only parts of the navmesh at a time. There are lots of paths you could find/see, but there are also some paths that couldn't be found/displayed. The results would still be useful but incomplete, and sometimes those missing pieces are what matters.


In this situation I would say that you need to use hierarchical pathfinding, as if you attempt to use a flat navmesh that takes more than several GB of memory to represent, you're in for a bad time.

If that's not your actual problem space, well, the advice still applies; you are looking at trying to brute-force a problem that almost assuredly has elegant algorithmic solutions that use far less resources and are amenable to streaming or other dynamically loaded setups.

Share this post


Link to post
Share on other sites
Also... Who are your users? If this is a quick-fix you're looking for to act as a band aid until a fundamentally better solution can be worked out, and if your users are internal or expected to have an IT budget, it could well be that the cheapest, most-expedient solution might be "buy more RAM" or "Rent some time on a big VM in the cloud".

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!