Sign in to follow this  

Memory management of disk-loaded resources

This topic is 3317 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

(I hope this is the right forum for this topic, apologies if it is not.) I am curious how people deal with the problem of loading resource files into memory, and dealing with not only the duplication of loading a compressed file into memory and having to decompress it also into memory, but also the problem of deciding which copies to keep in memory for future usage. Lets say your game stores all resource files in a ZIP archive. The game loads up, and you want to play a music loop in the background. You can memory-map the archive for easy reading, but you still need to decompress the background music file into a separate buffer to actually play it. So, the question becomes, do you deal with the entire music file sitting in memory, taking up space but making it convenient to loop over the entire song over and over as desired? Or do you just load and decompress a portion of the music file, streaming it in chunks as it plays, but ultimately requiring that you re-read and re-decompress the same chunks over and over as you loop to the beginning of the file each time? Though the first case sounds a lot easier, at some point the file becomes too big to keep in memory. But the music is supposed to still be playing in the background. What do you do then? Do games that use compressed archives really end up re-decompressing the same data over and over? Or is there another method I'm not thinking of?

Share this post


Link to post
Share on other sites
I would think they do indeed decompress it repeatedly, but just are very careful with what takes how long to decompress and what takes up how much memory. Memory management is a huge thing with games that have enough data going with them to warrant zipping it up. Also, decompressions is really rather fast so if it is done gradually or in small bits etc it wouldn't have much of an effect; MP3s, PNGs, most videos, all use compression. But, a little tiny mp3 player with few enough MHz to count using body parts can still decompress all of those fast enough to use. Zip compression is a bit more hefty but still not that slow.

Share this post


Link to post
Share on other sites
Quote:
Original post by Nairou

I am curious how people deal with the problem of loading resource files into memory, and dealing with not only the duplication of loading a compressed file into memory and having to decompress it also into memory, but also the problem of deciding which copies to keep in memory for future usage.


This duplication is a non-issue. If you're talking about PC, then you need one single buffer that is equal to size on disk. After file is decompressed, you can re-use this buffer for another decompression, or release it. The benefits of loading the file in bulk rather than reading it piece by piece will offer considerable performance improvements.

If you're on memory starved system, then you need to use some streaming algorithm to decompress.

mp3 for example is such. It has no structure, you just read in random bytes and search for frame header. Once you find it, you decode 1kb or so data for replay. Decoding is done each time as needed since it's so cheap.

Quote:
Lets say your game stores all resource files in a ZIP archive. The game loads up, and you want to play a music loop in the background. You can memory-map the archive for easy reading, but you still need to decompress the background music file into a separate buffer to actually play it.


Don't do that. Load entire file into memory.

If you use memory mapping, then you will get horrible performance since disk will be constantly trashing to read the same few kb over and over.

Either that, or your OS will cache the contents into file cache. So not only is your memory-mapped-no-memory-used file now using up RAM, it's also polluting the file cache.

And if memory does at some point become a problem, then you need to develop some form of streaming anyway. But not inside resources, but by de-allocating resources themselves. So if suddenly you find you can no longer keep 3 mp3s in memory, decide to stop playing one and unload it.

Share this post


Link to post
Share on other sites
The bottom line is that you must keep disk access to a minimum. It's bad enough on PC, but on a console it's even worse since in the worst case it means you're seeking an optical drive.

Whether or not you keep a compressed image in memory and decompress on demand, or just decompress the entire file initially, depends on the demands of your game. Decompression in general isn't really that slow, but obviously you don't want to spend cycles decompressing the same data over and over if you don't have to.

The key is to come up with a good working set algorithm that can manage which resources need to be resident in memory at any given time, and which ones can be streamed in/out as memory becomes available/unavailable. It can handle whether a certain resource should be resident in compressed form or decompressed on load. It can also handle different resource LOD levels -- should you load all mip-levels for a certain texture, or is there only enough memory/time to load the lowest mip-level? All this functionality eventually becomes part of a larger "streaming engine", which essentially handles resources being loaded in to memory from disk unloading resources from memory, only when absolutely necessary. It's difficult to write a good streaming engine, and just as hard to get rid of the glaring artifacts that result (texture popping for instance).

Share this post


Link to post
Share on other sites
Thanks for the replies, they have been very helpful. I can see the benefits of controlling disk access, and turning the resource manager into a stream engine of sorts is a very interesting idea.

I'm curious, given what's been discussed here, why do so many people tout memory-mapped files as being the best thing to use for all file access? I can certainly see the benefits of not having to allocate buffers and read chunks of the file, but if you lose performance by doing so, I don't know why you'd stick with it. Are there maybe certain limited cases when it is still preferred?

Share this post


Link to post
Share on other sites
Quote:
Original post by Nairou

I'm curious, given what's been discussed here, why do so many people tout memory-mapped files as being the best thing to use for all file access? I can certainly see the benefits of not having to allocate buffers and read chunks of the file, but if you lose performance by doing so, I don't know why you'd stick with it. Are there maybe certain limited cases when it is still preferred?


Memory management isn't just saying "I have 5MB available." There's also other issues to consider, such as continuous available memory.

Young'uns today don't remember the common occurrence back in the day: "This program requires 483kb RAM, only 479kb available". The problem was simple - you either had the memory, or you couldn't run the application. QEMM and tweaking config.sys was only way to get around.

It's not really a matter of preference, it's last resort to deal with a problem which would cause a complete failure.

When a file is memory mapped, OS simply treats it as part of virtual memory (swap file). I hope there is no need to mention what happens when OS starts paging and everything grinds to a halt. Now imagine your swap files were located on a CD. Biggest problem comes from the fact that disk is a sequential device, and memory mapping makes it look like random access. On SSD, things might even work ok.

Disks are good at streaming bulk continuous data. It's usually necessary to specify a few flags, perhaps defragment the file or organize data to match the sectors, but that is where disks will excel.

Disks are abysmal when you need to randomly access small pieces of data. This is why disk cache is crucial. It turns out, if you only need to read a few bytes, it's just as fast if OS reads 64k or even more around that location. Since many file operations will read data sequentially, the extra cost reading more data is almost 0 (disk is spinning in that direction anyway), but all subsequent reads will be done from memory (rather than requiring another disk seek, which are still around 10ms).

The key factor to realize here is that disk operations are so incredibly costly. It's not uncommon that reading compressed data and uncompressing it is faster than reading uncompressed data from disk. Assuming this data the undergoes same processing.

Also, music files are designed for streaming. You can just open file handle, and when you need more data, read it there. Memory mapping is primarily intended for random access (where it will suffer huge performance penalty).

And if we're talking PC, you typically have upwards of 1 Gig of RAM + 3-15Gb of virtual memory. That single extra buffer needed for one-time decompression isn't going to show up anywhere.

Share this post


Link to post
Share on other sites

This topic is 3317 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this