Jump to content
  • Advertisement
Sign in to follow this  
Extrarius

Automatically Paged File Mapping?

This topic is 4734 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

For a project I'm working on at work, I found the fastest way to deal with a huge file was to map the file to memory using the windows API. As the program reads the file, windows apparently loads new sections into RAM and also keeps old sections loaded. I really don't want windows to be paging out other things to keep the whole file in RAM when I'm only VERY rarely going to reexamine part of the file that's already been read. I see that in MapViewOfFile, I can manually specify that it should only load parts of the file, but is there any way to have it page it automatically to only use a certain maximum amount of RAM? Making such a change would take quite a while. Any ideas would be appreciated.

Share this post


Link to post
Share on other sites
Advertisement
I don't believe this is possible. Windows gets as good performance as it does by automatically caching mapped files, so it would be generally detrimental to allow otherwise. I should also mention that it only caches parts of the file that have been touched (either by read or write), with the possible exception of read-ahead caching. It shouldn't be too big of a deal.

Share this post


Link to post
Share on other sites
Depending on the type of mapping, mapping a file into memory should not use RAM.

Here's why:

If you have a shared mapping type (i.e. your changes can be seen by other processes, and are written back to disc), the OS only needs to keep parts of the file in memory which are being used.

It can page it back out, but it doesn't use up swap, instead it can just keep it in the file it's really mapped into.

Therefore, on a machine with little memory and no swap file, it should still be feasible in principle to map a very large file (say 500M) without a problem.

Of course file mappings are limited by address space (on 32bit machines), so you can't map more of a file into memory at once than the largest available block of address space.

If course OS supports VM overcommit, it will be possible to map in more (private, i.e. not file-backed) memory than is possibly available. Believe it or not, this is a feature, not a bug. Linux supports this (and it's sometimes enabled by default). It allows the memory manager to write cheques it can't cash, which sometimes enables it to perform better apparently.

I don't know whether the Windows VM supports overcommit (or if it's enabled).

Mark

Share this post


Link to post
Share on other sites
The file is mapped as read only, and windows IS only loading parts that are touched, BUT it keeps parts that have been touched in favor of other things that I want kept. It isn't much of a problem right now with the test files (60MB or so), but it will be worse later on the real files which are almost 1GB. Using up the address space is no problem, but it seems to like to actually keep whole file (from start to current read location since its mostly scanned sequentially) in RAM in preference to currently executing background processes, so they spend most of the time in page faults that force pagefile reads instead of actually doing their work.

Share this post


Link to post
Share on other sites
It shouldn't be a problem unless you read the whole file frequently (in which case you might want it to be cached, anyway...). I believe it's just a simple least-recently used algorithm. If the space is needed in the cache for other stuff, then the cached pages will get dumped. If you're really worried about it, you could try FILE_FLAG_SEQUENTIAL_SCAN on CreateFile, although I don't have proof that that flag affects file mapping caching.

Share this post


Link to post
Share on other sites
It's a problem because windows REALLY tries hard to keep the whole file in memory, and when RAM is 512MB and the file is 800 MB, it results in a system cache of a few megs (realy slowing down performance) and only a few meg of physical RAM available, so every time something else needs ram both programs slow to a crawl while windows pages things in and out one page at a time as neccessary.

Share this post


Link to post
Share on other sites
I've seen this happen. It is annoying to the end user when all their programs get paged out because of stuff like this (programs doing heavy disk I/O are the main culprits). That being said, I consider it an OS flaw in that it is too eager to throw away pages allocated to other programs.

Have you tried only partially mapping the file and then adjusting your view manually? Conceivably, you could have a class that wraps the mapped file and provides an overloaded operator[]. When you move outside the bounds of the current view, it would either extend the current view to include it, or start dropping pages off the back. I have no idea how slow this would be, but I don't see any other way from user space.

Share this post


Link to post
Share on other sites
Quote:
Original post by antareus
[...]Have you tried only partially mapping the file and then adjusting your view manually? Conceivably, you could have a class that wraps the mapped file and provides an overloaded operator[]. When you move outside the bounds of the current view, it would either extend the current view to include it, or start dropping pages off the back. I have no idea how slow this would be, but I don't see any other way from user space.
This will solve the problem, but it will also greatly complicate the code because I'll have to add all kinds of error detection that doesn't need to be there now (because once the file is mapped, nothing can go wrong with the mapping). I guess I'll have to implment this, but I was hoping there would be another option.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!