Resource memory management
Author
Okay, I'm designing the resource manager for my engine, and I've got some questions about memory management.
1. Before the resource manager can know what resources are available, it has to enumerate them. It uses somewhat of a virtual filesystem, composed of stock and mod files, so using real filesystem paths isn't an option. I assume this means it has to keep a list in memory of all available resource files, so it knows where to access them from? Seems wasteful, but I'm not sure what the alternative is.
2. Lets say the engine requests a resource file from the resource manager, like a texture. Besides finding and opening the file, how are the contents normally provided to the engine? Is the entire resource read into memory? Is the file usually just memory-mapped? Or does the engine specifically have to request pieces of the file to load? Resource files are usually access sequentially anyway, especially if they have to be fully loaded from an archive for decompression, but again it seems wasteful. Especially if, say, the engine loads a texture into memory, and then has to convert the texture into a format the rendering API can use (as an example, maybe this isn't true). Thats a lot of disk access and memory usage.
Is this how its done, or are there some fairly standard ways of accomplishing this? How do you do it?
1. For your VFS open, you can just search all paths/archives in order, or remember where each file is located, and use that when opening. It's a tradeoff between long search time and memory use.
As soon as you have multiple paths, the linear search gets rather slow, so I went with the second approach. Memory use is no problem: we have several thousand files; you only need to store one string->Location* std::map node per file, and one Location for each path or archive.
2. The majority of all files are read in one shot, so I provide a Load() routine on top of the normal file I/O calls. I've seen some engines using mmap throughout, but async I/O is faster (better throughput) when reading large files in one go.
It's possible to load on the fly (sequentially) from Zip archives.
Yes, you definitely want to avoid that. Allow all formats / inconveniences like upside-down images during development, and make the engine refuse them in release builds after writing a 'build' tool that converts everything to the optimal format.
HTH+HAND
As soon as you have multiple paths, the linear search gets rather slow, so I went with the second approach. Memory use is no problem: we have several thousand files; you only need to store one string->Location* std::map node per file, and one Location for each path or archive.
2. The majority of all files are read in one shot, so I provide a Load() routine on top of the normal file I/O calls. I've seen some engines using mmap throughout, but async I/O is faster (better throughput) when reading large files in one go.
Quote:Resource files are usually access sequentially anyway, especially if they have to be fully loaded from an archive for decompression
It's possible to load on the fly (sequentially) from Zip archives.
Quote:Especially if, say, the engine loads a texture into memory, and then has to convert the texture into a format the rendering API can use (as an example, maybe this isn't true). Thats a lot of disk access and memory usage.
Yes, you definitely want to avoid that. Allow all formats / inconveniences like upside-down images during development, and make the engine refuse them in release builds after writing a 'build' tool that converts everything to the optimal format.
HTH+HAND
As alwyas resource management is very interesting topic.
1. The alternative (actually, one of them) is to make your virtual file system mimic real pathnames and directory structure. Ie. you want to load texture foo.png from resource file Foo, you say: "TextureResMgr.Load("Foo\foo.png")" and it is resource mgr that is responsible for spliting pathname into names of modules, files etc. for loading those files, needed conversions, error handling etc.
And you can always use reference counting to prevent from keeping all resource files of memory: at program startup, load all resource file and enumerate all resources you find in them, then save that list somwhere (map?) and you can unload files. Then if someone want to access resource (through _full_ qualified pathname), you simply seek that file in enumerated ones, then if it's found, treat part to first \ as resource file name, load that file to memory, return appropriate resource AND set refCount of that file to 1. If you load additional resource from that file, simply ++refCount. If you release resource from that file, --refCount. If refCount == 0, you can safely unload that file couse there's nobody using it at the moment (of course assuming that your resource mgr is responsible for loading and deleting of resources and it can also 'freeze' unusued resources, so they aren't directly in memory but are loaded when there's need for them; else using references here isn't needed).
2. Well, in first point I've already written what is one of the alternatives. There's also one additional alternative that answers your 2 next questions:
The answer is simple: don't use resource files, store all your files directly inside directories ;-) and that's how I'm doing it in my actual project. But my case is very specific (developing _extremely_ open sourceAndOtherOpenStuff game) and your needs probably are different.
[edit] oops, Jan was faster :-]
1. The alternative (actually, one of them) is to make your virtual file system mimic real pathnames and directory structure. Ie. you want to load texture foo.png from resource file Foo, you say: "TextureResMgr.Load("Foo\foo.png")" and it is resource mgr that is responsible for spliting pathname into names of modules, files etc. for loading those files, needed conversions, error handling etc.
And you can always use reference counting to prevent from keeping all resource files of memory: at program startup, load all resource file and enumerate all resources you find in them, then save that list somwhere (map?) and you can unload files. Then if someone want to access resource (through _full_ qualified pathname), you simply seek that file in enumerated ones, then if it's found, treat part to first \ as resource file name, load that file to memory, return appropriate resource AND set refCount of that file to 1. If you load additional resource from that file, simply ++refCount. If you release resource from that file, --refCount. If refCount == 0, you can safely unload that file couse there's nobody using it at the moment (of course assuming that your resource mgr is responsible for loading and deleting of resources and it can also 'freeze' unusued resources, so they aren't directly in memory but are loaded when there's need for them; else using references here isn't needed).
2. Well, in first point I've already written what is one of the alternatives. There's also one additional alternative that answers your 2 next questions:
Quote:Is this how its done, or are there some fairly standard ways of accomplishing this? How do you do it?
The answer is simple: don't use resource files, store all your files directly inside directories ;-) and that's how I'm doing it in my actual project. But my case is very specific (developing _extremely_ open sourceAndOtherOpenStuff game) and your needs probably are different.
[edit] oops, Jan was faster :-]
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
