Resource/Asset Managment

Started by
14 comments, last by Tachikoma 12 years, 10 months ago
I'm just getting started on my game and I'm trying to find a consistent way of managing game assets, which I define to be things like textures, meshes, audio files, shaders, lua scripts, etc.

I'm looking to create something along the lines of a ResourceManager class, maybe templated, ie. typedef ResourceManager<Texture> TextureManager; but I'm not exactly sure of what the interface should be like.

I've searched around for some examples here, but none of them seemed like optimal solutions for me. Id like for it to reference count resources so when a particular mesh is no longer being used by anything, that asset is removed from memory. Although, it would probably be better that the resource not be removed immediately upon reaching 0 references, so maybe the ResourceManager keeps it's own reference so the count never drops bellow 1, and semi-periodically, it will find assets with 1 count and remove them. That way you don't have to re-load a file because it's count reached 0 but was needed again just a few seconds later. If a particular URI is requested which matches one of the assets already loaded, it will return a ref counted pointer to the asset already in memory, so you never have more than one instance of an asset loaded in memory. The system will also have to be thread and Async IO friendly, as I plan to use a task based system and asynchronous loading. The other issue I foresee, maybe, is serialization. I know storing pointers is bad for that, and a smart pointer would be no different.

Help with this would be greatly appreciated.
Advertisement
The interface comes naturally when you separate the concerns. To load resources, you need a ResourceLoader<>. You wrap this in a ResourceCache<>. Your cache might have a configurable CachePolicy, or you can use a hardcoded one for the moment. I use a simple cache that never unloads anything, instead each level uses its own cache. But to avoid loading resources that are already in memory during level transitions I have a resource loader that searches in an existing cache first before hitting the disk.

Serialisation can be done by storing the URI's used to load the resource (or a mapping of numberic ID -> URI to save space) but I don't think that is part of the interface of either of these classes.

Beware "Manager" classes, their scope is not well defined and you end up with a bunch of logic that is only vaguely related at best.
Hmm, I'm not clear on what functionality ResourceLoader and ResourceCache will serve. Could you maybe give and example of how it would be used?

I'm also trying to figure out how to handle the asynchronous loading. What happens when the renderer tries to use a texture that is still being loaded?
After the resource management has detected that the resource ID of a request cannot be resolved to a resource data object, it is responsible to trigger the resource data supply. As a consequence the resource data will become available to the requesting client. Over time the memory consumption may grow above the limits of what is allowed or desired. A reference counting allows to remove the resource data from memory as soon as it is detected as no longer needed. But there is no guarantee that the same resource data is not needed in few seconds again (e.g. when the player's avatar is moved from a room to the next, and leaving a room kills references). Hence unloading resource data can be deferred not only until the management detects the reference counter hit 0, but until memory is detected to become low, or resource data is detected to become outdated by other means, or whatever. This keeping of resource data longer then the explicitly known duration of use is called "resource caching". Deciding which resource data to unload is called the "cache policy". E.g. unloading the resource data with the oldest timestamp of loading or reference count 0 are some possibilities.

Above I've written "to trigger the resource data supply". This targets the fact that resource data need not necessarily be loaded to come to life. Computation of resource data is also a possibility. E.g. a mirror may show a texture that is rendered just-in-time (render-to-texture approach), or the sky shows a cloud texture with slow variation over time (done by texture synthesis). However, IMHO resource loading should not be a compelling part of the resource management, although it is usually the most important kind of supply.

However, when a supply suitable for loading is triggered, things like virtual file systems come into play. I.e. loading need not necessarily mean loading of the content of a individual file, but perhaps loading from the content of an archive. It also may mean to investigate a couple of directories in order (a.k.a. search paths) where the first hit of matching data is loaded. In summary, separating resource loading from resource holding is a good thing.

Perhaps you've noticed that I've written "resource data objects" above. That is because one can think of distinguishing resource objects from resource data (objects). Here resource objects are meant to allow a finer control of resource management, what usually mean that resource objects will live longer than resource data. In resource objects a local policy of loading (e.g. pre-loading of resource data when the game or the level is started) or a local policy of caching (e.g. never unload the belonging data until the level is completed) can be stored, or the resolved path for loading can be cached. In the case of threaded loading, the resource object can be used to track the loading state, and to hold the pointer to a delegate, etc.

Hmm, I'm not clear on what functionality ResourceLoader and ResourceCache will serve.
[/quote]
ResourceLoader -> loads resources
ResourceCache -> caches resoures

Could you maybe give and example of how it would be used?
[/quote]
A simple example would be:

#include <map>
#include <string>
#include <memory>
#include <iostream>

// Simple file name based resource loader.
template<typename T>
class ResourceLoader
{
public:
ResourceLoader(const std::string &basePath)
:
basePath(basePath)
{
}

std::shared_ptr<T> load(const std::string &name)
{
return std::make_shared<T>(basePath + name);
}

private:
std::string basePath;
};

template<typename T>
class ResourceCache
{
typedef std::shared_ptr<T> ResourcePtr;
typedef std::map<std::string, ResourcePtr> ResourceMap;
public:
ResourceCache(ResourceLoader<T> *loader)
:
loader(loader)
{
}

ResourcePtr get(const std::string &name)
{
ResourceMap::iterator i = map.find(name);
if(i != map.end())
{
return i->second;
}
ResourcePtr resource = loader->load(name);
map.insert(std::make_pair(name, resource));
return resource;
}

private:
ResourceMap map;
ResourceLoader<T> *loader
};

class Example
{
public:
Example(const std::string &path)
:
path(path)
{
std::cout << "Example resource construction: " << path << '\n';
}

void print()
{
std::cout << "Example resource use " << path << '\n';
}

private:
std::string path;
};

int main()
{
ResourceLoader<Example> loader("resources/examples/");
ResourceCache<Example> cache(&loader);

std::shared_ptr<Example> example;

example = cache.get("hello.example");
example->print();

example = cache.get("goodbye.example");
example->print();

example = cache.get("hello.example");
example->print();
}

Asynchronous loading can be achieved by having some kind of "Future" resource. That is, these resources would either block on first use (bad - might as well use synchronous loading) or be no-ops until loaded (e.g. use a transparent 1x1 texture until the actual texture is fully ready, a silent sound, an empty mesh). The other alternative is to have your resource loader return a third state (instead of NULL/exception and Resource, it can return "pending"). The game logic keeps trying to load the resource until it gets a non-pending result.

The main problem with the asynchronous loading is where do you want to deal with it. My former proposal forces the logic into the loadee, the client code gets a proxy it can use immediately, whereas the latter proposal puts the burden on the client logic but makes the resource implementation simpler.

What kind of game are you writing? Try to keep your resource management code in line with the scope of the game. Not every game needs asynchronous loading, and simpler games might benefit from a greedy approach of simply loading all the resources in a particular directory tree once at startup, if memory allows.
Using a future had crossed my mind, but it didn't seem appropriate for the reason you mentioned. I also thought about returning a default resource, but my question with that is how to swap out the default with the real resource as soon as it becomes available.

The game is a FPS, and the reason I want async loading is because I'm opposed to loading screens. I plan for seamless transitions from one area to the next, so there aren't really "levels" per se. Some assets will be used throughout the game, and others will come and go.
Don't return a default resource, return a proxy that binds a default resource until the actual resource is loaded.
I also have to take into account resources that don't come from disk. Procedural textures and render-to-textures, as haegar mentioned. I would also like to incorporate the ability for resources loaded from files to automatically reload themselves if they receive a notification from the file system that the file has been modified.
rip-off : What is the purpose of ResourceLoader, if all it does is construct a new T using T's constructor?

rip-off : What is the purpose of ResourceLoader, if all it does is construct a new T using T's constructor?
[/quote]
It is a separation of concerns, I don't think a cache should be responsible for creating the objects it is caching. Its job is to act as a buffer between the client and the "expensive operation", whatever that happens to be.

The resource loader could do other things, it could do the procedural generation, or load from an archive. Perhaps it needs to be like a factory, e.g. the texture needs to be created in by a particular Renderer instance. Or maybe the resources don't do I/O themselves, they expect to be constructed with in-memory data - the loader would do the I/O and pass the byte buffer to the Resource constuctor.

I gave my example one a "basePath", which was supposed to hint at this idea. Maybe I didn't make it clear enough. But it doesn't just create a new T, simple as it is.


I also have to take into account resources that don't come from disk. Procedural textures and render-to-textures, as haegar mentioned. I would also like to incorporate the ability for resources loaded from files to automatically reload themselves if they receive a notification from the file system that the file has been modified.
[/quote]
A proxy can do all of these things. Monitoring for changes is similar to asynchronous loading, your proxy keeps rendering with the old value until the new values is created, then it can just change which value is bound. Procedural generation can be handled similarly, conceptually it is no different from asynchronous loading from disk.

For Render-To-Texture, you might add a function to the cache so that you can add a named object to it, if you want to cache such textures. If not - then it isn't really a concern of your loading/caching infrastructure, it is something else.

The main point really isn't to give you some concrete design saying "this is the interface for managing assets", but rather to show that if you split the functionality into a number of distinct classes with well designed responsibilities it makes the code easier to manage. I mean, if you look at the resource cache code I gave you can see that it doesn't contain any bugs (apart from not maintaining an invariant that the loader is not-null). If the code for the caching, creating, procedural generation, background loading, swapping default/updated resources as I/O completes were all in one monolithic "ResourceManager" class then I think it would be much less clear what is going on.

You were looking for an interface for such a class, all that I am suggesting is breaking the "management" of resources into distinct sub units of functionality. These may interact, but at least each sub-unit has a reasonably well defined scope of operations.

This topic is closed to new replies.

Advertisement