I'm writing a very small engine to learn more about modern OpenGL, and i'm looking for some examples and learning material on how to write an efficient resource manager for assets like textures, models, sounds, etc. Does anyone recommend anything? Books, tutorials, samples, libraries, anything. :)
Efficient resource manager for OpenGL in C++?
Efficient for the computer or for you? What a about put files in folders and load everything on startup into RAM? When (If) you ever hit 1 GB your objectOrientenDependencyInjectedEntityMappedCode is surely flexible enough to get a streaming treatment.
I just store them in a vector, and pass the array index around inside an opaque handle ¯\_(?)_/¯ Also keep a map around for string to index lookups. Right now indexes are never reused, makes it easier to track use-after-delete.
Of course things get rather a lot more complex with streaming.
I recently designed a resource management system. I liked the straight-forwardness of DOOM WAD files, so I copied this style and made my own solution. Maybe you'll like the idea too:
- Store all assets within a single archive file (does not need to be encrypted)
- Beginning of file is a table that maps file names to offsets in the archive
- Each file-entry is just the raw file placed straight into the archive, can do compression, but I didn't
The nice thing about this style is when the time comes to package the game up for release there's only one asset file. It can be a little weird if the file gets over 2GB in size since your file i/o API will need to be able to handle 64-bit integers for file sizes. If disk-seek optimization is ever needed (or if you want to do seek optimization for fun) the ordering in the archive can be modified. When the game launches the archive can opened and the map-table can be read into game memory -- this can act like a "symbol table" of all the files that the game can request. The symbols are stored in a hash table, where the key is the file-path and the value is information about that asset + a handle to the asset (if it's been loaded into memory).
To let artists keep modifying files at-will all the asset files are kept in our repository and there's a function that can be called to package up all the assets into a new archive. File name + paths are used as keys into the archive, and placed in that initial lookup-table I described earlier. This way the asset archive doesn't really affect the workflow of the artists. We used some recursive directory-crawling code to do this.
Anyway, hope that helps!
Yeah why you need a manager for something which is plain data? When I read code from others, "Manager" typically marks the classes which have to be replaced by a sane design pattern.
Since you said "small", my approach might just work for you.
I'm simply memory mapping a large binary file which is "compiled" with a custom tool. When I say "compiled", it basically means reading a structured "definition file" that tells the compiler who references whom and what the asset names and file names are. It calculates a hash of every name prepended with a salt and then checks that all hashes are unique. Then it calculates offsets by adding up sizes, and pukes everything (array of structures, followed by binary contents of every file) into one big, ugly, blob. Some people sneer at storing a structure as-is, and they are kind of right, but with asset compiler and game built with the same compiler and running on the same architecture, it's a bit of a "so what" situation.
For the few assets that you actually want to look up by name later (in fact, other than "load screen", and "menu", and "level 1", you shouldn't need all that many), you must know the salt, and unluckily, you must do the hash at runtime, so the salt needs to be stored in the file header. For the overwhelming majority of assets (which are referenced by other assets within the asset file) you don't really care about their name (it would of course be possible to load them with a string name, but why... if level 1 uses assets #123 and #456, that's really all you need to know). Each asset that references another one does that by its hash. The asset compiler makes sure, at build time, that all the hashes are all consistent. It would arguably be "better" do do it with strings (more explicit, anyway), but I'm pretty happy with this "mostly string free" approach.
Note that if you plan to frequently add assets, my "mostly string free" approach is not truly the best strategy (you could even say "catastrophic"). During development, unless your assets are terabyte-scale, the rebuild is only mildly annoying. It's more an issue if you intend to do a lot of changes regularly after shipping (say, you want to add some downloadable contents every week). In that case, you probably want a more complicated approach anyway, though. You most probably want to consider moving the "table of contents" into a separate file, or alternatively at the end of the datafile, with an offset to the start of the TOC in the file header (otherwise, adding assets is a nightmare either way because you must move stuff around).
Now, regardless of how you identify assets, by string or by hash, (that's actually pretty inconsequential!) when the game starts, you memory map that entire ugly pig of a file. As long as your assets alltogether aren't around a gigabyte or so, this is no problem (you said "small", right?) even on 32 bit. On 64 bit it's no issue anyway.
The actual "asset manager" consists basically of finding the resource's hash value (I don't even bother with something overly intelligent, simple linear search -- works better than you think, at 16 bytes per entry it's not a terribly large amount of memory you need to walk over) and then adding the offset to the mapped file's pointer. That's it, the operating system does the rest, and it does it pretty good.
Allocate? Do nothing. Free? Do nothing. Manage? Do nothing. Just dereference the pointer when you feel like doing it.
If you want to be extra zealous, pre-fault pages that you intend to access with a worker thread, that way you gain a little speed (but not much, readahead works surprisingly well). The main reason to do the faulting in a dedicated thread would be in order not to block the render thread and have hiccups, if you also load assets while the game is already running (as opposed to "at start" or "at level load"). If you do something more elaborate (like multithreaded loading, maybe with decompression) that isn't an issue anyway, the worker threads will do the faulting without you even knowing. Render thread consumes what the decompress thread produced whenever the condvar says "done".
I wouldn't ever want to use anything but memory mapping again. As long as you don't run into address space trouble under 32 bits, it reduces a pretty complex problem to "yeah, you know, just don't worry". With either workers in place doing decompression and stuff, or a dedicated prefaulter thread, it runs really really nicely, fast, and apart from the single syscall for mapping, it is 100% portable and system-agnostic.
Windows overlapped I/O traditionally used to be the often recommended "go to solution", but apart from being totally un-portable, the code is also way more complex, and at least on any system that I've seen, it's substantially slower than memory mapping, too. AIO doesn't work nearly as well under Linux either (you could even say "terribly").
Plus, with pretty much every AIO/overlapped approach you are completely tossing the filesystem cache over board, which is an immensely bad thing to do. It means you basically need to rebuild a sort of caching mechanism in your application although one that works perfectly well already exists in the operating system. Yes, you might be able to squeeze out 0.5% more performance. Maybe. Or maybe not. You might do a lot of redundant work in order to run 50% slower in the end, too.
If you want to compress assets on disk, it gets quite a bit more complicated. In that case, you must allocate and actually manage memory because you need some place where you can store the decompressed data. Enter handles or refcounted pointers.
I personally don't think nowadays compression is really that much worthwhile any more (but your mileage may certainly differ). You still gain a little if you use the correct algorithm (think e.g. LZ4, which is basically as expensive as two memcpy calls) but some are the same speed or slower than a modern harddisk, and considerably slower than a SSD. For example, zlib is just about half the speed of what the SSD delivers on my computer, and lzma is considerably slower than what even a traditional harddisk would do in bulk transfer.
So, for "small" (i.e. not something that will need a bluray), I think skipping compression alltogether may be an entirely feasible option. Just... memory map, and use the pointer.
I'm simply memory mapping a large binary file which is "compiled" with a custom tool. When I say "compiled", it basically means reading a structured "definition file" that tells the compiler who references whom and what the asset names and file names are. It calculates a hash of every name prepended with a salt and then checks that all hashes are unique. Then it calculates offsets by adding up sizes, and pukes everything (array of structures, followed by binary contents of every file) into one big, ugly, blob. Some people sneer at storing a structure as-is, and they are kind of right, but with asset compiler and game built with the same compiler and running on the same architecture, it's a bit of a "so what" situation.
For the few assets that you actually want to look up by name later (in fact, other than "load screen", and "menu", and "level 1", you shouldn't need all that many), you must know the salt, and unluckily, you must do the hash at runtime, so the salt needs to be stored in the file header. For the overwhelming majority of assets (which are referenced by other assets within the asset file) you don't really care about their name (it would of course be possible to load them with a string name, but why... if level 1 uses assets #123 and #456, that's really all you need to know). Each asset that references another one does that by its hash. The asset compiler makes sure, at build time, that all the hashes are all consistent. It would arguably be "better" do do it with strings (more explicit, anyway), but I'm pretty happy with this "mostly string free" approach.
Note that if you plan to frequently add assets, my "mostly string free" approach is not truly the best strategy (you could even say "catastrophic"). During development, unless your assets are terabyte-scale, the rebuild is only mildly annoying. It's more an issue if you intend to do a lot of changes regularly after shipping (say, you want to add some downloadable contents every week). In that case, you probably want a more complicated approach anyway, though. You most probably want to consider moving the "table of contents" into a separate file, or alternatively at the end of the datafile, with an offset to the start of the TOC in the file header (otherwise, adding assets is a nightmare either way because you must move stuff around).
Now, regardless of how you identify assets, by string or by hash, (that's actually pretty inconsequential!) when the game starts, you memory map that entire ugly pig of a file. As long as your assets alltogether aren't around a gigabyte or so, this is no problem (you said "small", right?) even on 32 bit. On 64 bit it's no issue anyway.
The actual "asset manager" consists basically of finding the resource's hash value (I don't even bother with something overly intelligent, simple linear search -- works better than you think, at 16 bytes per entry it's not a terribly large amount of memory you need to walk over) and then adding the offset to the mapped file's pointer. That's it, the operating system does the rest, and it does it pretty good.
Allocate? Do nothing. Free? Do nothing. Manage? Do nothing. Just dereference the pointer when you feel like doing it.
If you want to be extra zealous, pre-fault pages that you intend to access with a worker thread, that way you gain a little speed (but not much, readahead works surprisingly well). The main reason to do the faulting in a dedicated thread would be in order not to block the render thread and have hiccups, if you also load assets while the game is already running (as opposed to "at start" or "at level load"). If you do something more elaborate (like multithreaded loading, maybe with decompression) that isn't an issue anyway, the worker threads will do the faulting without you even knowing. Render thread consumes what the decompress thread produced whenever the condvar says "done".
I wouldn't ever want to use anything but memory mapping again. As long as you don't run into address space trouble under 32 bits, it reduces a pretty complex problem to "yeah, you know, just don't worry". With either workers in place doing decompression and stuff, or a dedicated prefaulter thread, it runs really really nicely, fast, and apart from the single syscall for mapping, it is 100% portable and system-agnostic.
Windows overlapped I/O traditionally used to be the often recommended "go to solution", but apart from being totally un-portable, the code is also way more complex, and at least on any system that I've seen, it's substantially slower than memory mapping, too. AIO doesn't work nearly as well under Linux either (you could even say "terribly").
Plus, with pretty much every AIO/overlapped approach you are completely tossing the filesystem cache over board, which is an immensely bad thing to do. It means you basically need to rebuild a sort of caching mechanism in your application although one that works perfectly well already exists in the operating system. Yes, you might be able to squeeze out 0.5% more performance. Maybe. Or maybe not. You might do a lot of redundant work in order to run 50% slower in the end, too.
If you want to compress assets on disk, it gets quite a bit more complicated. In that case, you must allocate and actually manage memory because you need some place where you can store the decompressed data. Enter handles or refcounted pointers.
I personally don't think nowadays compression is really that much worthwhile any more (but your mileage may certainly differ). You still gain a little if you use the correct algorithm (think e.g. LZ4, which is basically as expensive as two memcpy calls) but some are the same speed or slower than a modern harddisk, and considerably slower than a SSD. For example, zlib is just about half the speed of what the SSD delivers on my computer, and lzma is considerably slower than what even a traditional harddisk would do in bulk transfer.
So, for "small" (i.e. not something that will need a bluray), I think skipping compression alltogether may be an entirely feasible option. Just... memory map, and use the pointer.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
