# Resource Manager Design

## Recommended Posts

Armagedon    1234

Hello,

few days ago i started writing my own Resource Manager. It's not my first attemp to do this so i'm trying to design it as good as possible, but i run to couple of issues.

So i've got Resource<T> class that takes as template parameter some data.

For example if i want to pack Shader inside it , it's done like this:

typedef Resource<ShaderData> Shader; //Somewhere in code


But the problem is in ResourceManager, to make it most flexible i decided to make ResourceLoaders as Depedency Injection objects.

Let's say:

class ResourceManager
{
public:

private:
}

{
public:
}

{
public:
{
}
}


The problem is that i have no idea how to workaround problem with additionall parameters for given loaders.

I've come with ideas of using va_list, initializer_list and even variadic_templates but they're not really beautiful solutions.

Here is my question: How to design(or redisign) current way of managing those loaders with custom number of parameters?

Thanks for any help :)

##### Share on other sites
Irlan    4067

So i've got Resource class that takes as template parameter some data.

Each type of resource should have a manager. Trying to abstracting everything it's a waste of production time; you can even got something, but at the end you'll end up with a more complicated and unmantainable architecture.

In your case, you can have a templated map class that delete all pointers when it's time to shutdown the the application. Example from my old project:

#ifndef MAP_H_
#define MAP_H_

#include <map>

template< typename _tKey, typename _tVal >
class CMap {
public :
inline ~CMap() { Reset(); }

inline void Reset() {
for (std::map<_tKey, _tVal>::iterator I = m_mValMap.begin();
I != m_mValMap.end();
++I) {
delete I->second;
}
m_mValMap.clear();
}

inline _tVal Get(const _tKey& _kKey) const {
std::map<_tKey, _tVal>::const_iterator I = m_mValMap.find(_kKey);
return I != m_mValMap.end() ? I->second : NULL;
}

inline void Add(const _tKey& _kKey, _tVal _vVal) {
if (_vVal) { m_mValMap[_kKey] = _vVal; }
}

inline void Remove( const _tKey& _kKey ) {
_tVal* ptVal = Get(_kKey);
if ( ptVal ) {
delete ptVal;
m_tMap.erase(_kKey);
}
}
protected :
std::map<_tKey, _tVal> m_mValMap;
};

#endif


...and on each resource manager, you can have a map of each base resource class.

But the problem is in ResourceManager, to make it most flexible i decided to make ResourceLoaders as Depedency Injection objects.

You can explicitely tell the correspondent resource loader to request other resources as a dependency. Example:

CTextureManager uses a CImageManager.

Pass the image manager to request a image and create a texture from that.

I've come with ideas of using va_list, initializer_list and even variadic_templates but they're not really beautiful solutions.
Here is my question: How to design(or redisign) current way of managing those loaders with custom number of parameters?

Would be better separating these responsabilities for each type of resource. Each resource can be inherited from a base class. Example:

A texture can be a two-dimensional texture or a render target, but both resources need to be managed differently even if they're directly related. So, you should create a texture manager and a render target manager, even if your texture it's inherited from a render target. A texture manager requests images from the image loader, so you'll need to explicity define that, passing a image class pointer to the texture manager (if you're using a instance based architeture) or requesting directly the image from the image manger.

Edited by Irlan

##### Share on other sites
Juliean    7077

How about just using a variadic paramter pack? In the simpliest way, you can just use a map<std::string>, where you store a variant-class, like boost::variant.

class ResourceLoader
{
public:
virtual any load(void* Data, const std::map<std::string, boost::variant>& Params) = 0; //Loads given from memory data
}

{
public:
any load(void* Data, const std::map<std::string, boost::variant>& Params)  //Here is problem, loader would like to know also about Width, Height etc.
{
boost::variant& Size = Params["Size"];
}
}


You could typedef the map or wrap it in a class, but the concept is the same. You'd also have to lookup how boost::variant (or the variant class of your choice) works.

Edited by Juliean

##### Share on other sites
Strewya    2798

class ResourceLoader
{
public:
virtual any load(void* Data, void* Params) = 0; //Loads given from memory data
}

struct TextureParams
{
int width;
int height;
//and whatever extra info you want
};

{
public:
any load(void* Data, void* Params)  //Here is problem, loader would like to know also about Width, Height etc.
{
TextureParams* textureParams = (TextureParams*)Params;
textureParams->width;
textureParams->height;
//...
}
}


##### Share on other sites
jmakitalo    668

Each type of resource should have a manager. Trying to abstracting everything it's a waste of production time; you can even got something, but at the end you'll end up with a more complicated and unmantainable architecture.

I strongly agree. Storing inhomogeneous data typesafely and without unnecessary typecasting is difficult in C++. The most natural way is to make one manager per type of resource. This also works in other similar problems, such as in entity component schemes (one manager per component).

You may want to check my recent thread http://www.gamedev.net/topic/663222-resource-manager-for-open-world-game/ . I'm still in the process of making a generic resource manager framework myself.

Depending on your requirements, you may want to consider the use of handles to mediate resource usage.

I discourage this approach. It's the "C way" and is not typesafe.

##### Share on other sites
Diventurer    1631

I made a generic resource manager, and I find that it's working quite well.

I have an XML file where you define the resources, and you can add custom properties to your resource class.

<Container name="sprites">
<Container name="player">
<Resource class="Sprite" name="run" path="run.png" frames="8"/>
</Container>
</Container>


You need to override the initialise() and load() functions:

void Sprite::initialiseParameterTemplate() {
setParameter("frames", 1);
setParameter("class", "Sprite");
}

void Sprite::load() {
m_texture = new Texture();
m_texture->setParameter("path", path());
}


But after you've got that done, it's really just plug and play.

addResourceClass("Sprite", [](){ return new Sprite(); });
Sprite* sprite = resource<Sprite>("sprites/player/run");
int frames = sprite->frames(); // 8 - it returns "sprite->getParameter("frames").Integer" 

Is there anything with this design that could really make things break later on?

I hear so much bad about going this way, but I feel it's just perfect.

Edited by irbaboon

##### Share on other sites
Strewya    2798

I discourage this approach. It's the "C way" and is not typesafe.

A void* works. I also don't like it that much, but without going full template mode, it's the most simple solution to the problem the OP presented with minimal code impact.

IMO, a better way would be to stop trying to homogenize/generalize resource management until you have written non-generic managers for every resource type you're using, and have stopped adding additional code to them. Only then can you safely say "ok, these have a lot in common, maybe i can generalize this part a bit". But by then, you have working managers, so what's the point

I hear so much bad about going this way, but I feel it's just perfect.

Try adding sounds, scripts, entire spritesheets, texture atlases, meshes, fonts, etc. Write dummy code just to prove to yourself it doesn't break.

##### Share on other sites
Diventurer    1631

I hear so much bad about going this way, but I feel it's just perfect.

Try adding sounds, scripts, entire spritesheets, texture atlases, meshes, fonts, etc. Write dummy code just to prove to yourself it doesn't break.

Yeah, I've got several classes in my library I've already tested with: sounds, music, UI style files, textures, sprites, fonts, meshes, shaders..

And I also have made a game level as a resource in a game, and it worked as expected.

I'm just curious why people say it's bad.

##### Share on other sites
Diventurer    1631

I know this is controversial, but to be honest, that just makes it a lot more complicated than it needs to be.

It is probably because I'm still a novice, but I just feel adding tons of small classes do nothing more than make everything messy.

Also, all my resource manager actually does is:

* parse the XML file

* call the "load()" function of a resource that is going to be loaded

* get resource by name from a vector of ResourceContainer which has a vector of Resource.

Sorry for being stubborn, but I need some serious convincing to see the faults of this.

##### Share on other sites
jmakitalo    668

Is there anything with this design that could really make things break later on?

I hear so much bad about going this way, but I feel it's just perfect.

If I understand correctly, the resources derive from a base class that stores parameters as some name-value pairs.

I have also been thinking a lot how to associate parameters with resources so that they could be easily read from/written to files and perhaps used to automatically generate in-game menus for editing. However, storing the name literals is wasteful. For example, probably most "sprites" contain "frames" parameter. Each resource then stores this same literal for nothing. Also, searching parameters by name is slow, so the values should be cached in actual variables for run-time use.

I think I found a rather good way to overcome these issues. I only store parameters as proper typed variables. For bulk actions (serialization etc.), the resource creates a collection of generic variable objects, which act as mediators. For example:

class CSprite
{
private:
int frames;
std::string name;

public:
std::vector<CVar> getParameters()
{
std::vector<CVar> p;

p.push_back(CVar("name", &name));
p.push_back(CVar("frames", &frames));

return p;
}
};


The generic variables (type CVar) store pointers to the actual data. To construct a resource from a datafile, the getParameters() is called to get a routing to the data. These parameters are only used temporarily. My actual implementation uses a base class CVarBase for CVar<T> and vectors of base class pointers are returned, so this was a simlified example.

##### Share on other sites
haegarr    7372

I know this is controversial, but to be honest, that just makes it a lot more complicated than it needs to be.
It is probably because I'm still a novice, but I just feel adding tons of small classes do nothing more than make everything messy.

[...]

Sorry for being stubborn, but I need some serious convincing to see the faults of this.

The Sprite class, as an example, defines how a sprite is represented when being loaded into working memory, i.e. it manages sprite typical data for runtime purposes. So far so good. Now you add the possibility to load the data from a XML fragment, so that the Sprite class can construct its own instances. Okay, but we need to generate Sprite instances at runtime as well, so give it another factory method for this, too. If I have that, I could generate sprites by code and save them for later reload, so having a save method would be nice. Well, I want to support reading from and writing to files, but reading from and writing to memory would make networking more convenient, so methods for that are fine as well. Hmm, now that levels get bigger, I want to support binary data files as well as XML files. Oh wait, now that my Sprite class has this new fantastic feature, but existing files have not, I need to support a second generation of routines, and, since I want to support older engines, perhaps also saving routines. Now just putting in the render routines for OpenGL 3 for older machines and those for OpenGL 4 would complete it mostly. Support for mouse picking, because of the editor I'm planning, is a must of course. A bit of collision detection, and ...

Although exaggerated, the story above is what happens in reality. To defeat this from the very beginning, the single responsibility principle was defined. Sure, at the moment you say "I have only 3 responsibilities in my manager class, that is still maintainable", and you're right. However, this changes with time, and it always changes in the wrong direction if you not defeat it explicitly.

Notice that this does not mean that there is not something like a manager. However, such a manager would be a facade class, where clients find a concentrated API, but the work is done behind the facade by dedicated objects then, just controlled by the manager.

In the end, SRP simplifies the ability to exchange parts (for example the implementation of the resource cache), and to develop and test aspects separated (for example the versioning of resources on mass storage), and to still understand a piece of software after half a year or when being developed by another person, and to re-use it in another context as well. Its advantage is found in the mid to long run.

##### Share on other sites
Armagedon    1234

Propably i will go with Juliean way, due to the fact it's fastest way without redesigning current design.

But, Irlan idea is really cool. I know that i need to create own manager for each resources (i just call them loaders).

ResourceManager is intended to be a Facade just to don't have pass multiple managers between classes (yeah i know that the god objects are anti-pattern but in this case it's easier and clearer than having to store everywhere multiple managers).

##### Share on other sites
Diventurer    1631

Is there anything with this design that could really make things break later on?

I hear so much bad about going this way, but I feel it's just perfect.

If I understand correctly, the resources derive from a base class that stores parameters as some name-value pairs.

I have also been thinking a lot how to associate parameters with resources so that they could be easily read from/written to files and perhaps used to automatically generate in-game menus for editing. However, storing the name literals is wasteful. For example, probably most "sprites" contain "frames" parameter. Each resource then stores this same literal for nothing. Also, searching parameters by name is slow, so the values should be cached in actual variables for run-time use.

Nono, I don't store the frames parameter in Resource.

This map is defined in Resource:

std::map<std::string, Parameter> m_parameters;

Then each class that derive from Resource, needs to implement an initialise() function which has to at least do one thing to work with the ResourceManager: set the "class" to the class name. Read my other reply again.

Either way; I disagree that it's wasteful, because it's not like you'll have thousands of resources loaded at once. Even then, it's not that bad. We're talking some kilobytes in total.

I'll agree there probably is a better way to do it though.

I know this is controversial, but to be honest, that just makes it a lot more complicated than it needs to be.
It is probably because I'm still a novice, but I just feel adding tons of small classes do nothing more than make everything messy.

[...]

Sorry for being stubborn, but I need some serious convincing to see the faults of this.

The Sprite class, as an example, defines how a sprite is represented when being loaded into working memory, i.e. it manages sprite typical data for runtime purposes. So far so good. Now you add the possibility to load the data from a XML fragment, so that the Sprite class can construct its own instances. Okay, but we need to generate Sprite instances at runtime as well, so give it another factory method for this, too. If I have that, I could generate sprites by code and save them for later reload, so having a save method would be nice. Well, I want to support reading from and writing to files, but reading from and writing to memory would make networking more convenient, so methods for that are fine as well. Hmm, now that levels get bigger, I want to support binary data files as well as XML files. Oh wait, now that my Sprite class has this new fantastic feature, but existing files have not, I need to support a second generation of routines, and, since I want to support older engines, perhaps also saving routines. Now just putting in the render routines for OpenGL 3 for older machines and those for OpenGL 4 would complete it mostly. Support for mouse picking, because of the editor I'm planning, is a must of course. A bit of collision detection, and ...

Although exaggerated, the story above is what happens in reality. To defeat this from the very beginning, the single responsibility principle was defined. Sure, at the moment you say "I have only 3 responsibilities in my manager class, that is still maintainable", and you're right. However, this changes with time, and it always changes in the wrong direction if you not defeat it explicitly.

Notice that this does not mean that there is not something like a manager. However, such a manager would be a facade class, where clients find a concentrated API, but the work is done behind the facade by dedicated objects then, just controlled by the manager.

In the end, SRP simplifies the ability to exchange parts (for example the implementation of the resource cache), and to develop and test aspects separated (for example the versioning of resources on mass storage), and to still understand a piece of software after half a year or when being developed by another person, and to re-use it in another context as well. Its advantage is found in the mid to long run.

This is probably going to sound stupid, but I'm going to ask specific questions so I can get some confusion cleared up:

2. Should ResourceManager have all the ResourceContainers (which then contain Resource objects)?

3. Do you mean that the Sprite class should not have functions to write/read buffers? If so, do you mean there should be a SpriteSerializer class?

Also, this post was a bit longer, but I realised at least one of your points while writing it. I'm still curious what your reply is though.

##### Share on other sites
jmakitalo    668

Nono, I don't store the frames parameter in Resource.

This map is defined in Resource:

std::map m_parameters;

Then each class that derive from Resource, needs to implement an initialise() function which has to at least do one thing to work with the ResourceManager: set the "class" to the class name. Read my other reply again.

Either way; I disagree that it's wasteful, because it's not like you'll have thousands of resources loaded at once. Even then, it's not that bad. We're talking some kilobytes in total.

I'll agree there probably is a better way to do it though.

Ok, replying to this goes somewhat off topic, but this is very timely for me.

So you do store the parameters as name-value pairs like I suggested. It's a map. I guess that the Parameter class stores some union of admissible parameter types? Anyway, if you don't cache the values to some C++ types, each time you request a parameter from a resource you will need to make O(logN) lookup of the map, performing string comparisons. This may be neglible in some simple 2D game with a few resources, but I would not use it for rendering complex 3D scenes with hundreds of resources (with multitude of parameters) accessed each frame.

Storing the names may not take more than a few kilobytes, but usually such degeneracy suggests that the design could be improved.

Edited by jmakitalo

##### Share on other sites
haegarr    7372

First of, what we discuss here is the OOP way of doing things, and that in a manner with respect to the mid to long run. It is just an approach to defeat problems that may occur in the future when software should be extended or reworked in some way. There is no constraint to follow it absolutely; software worked also before OOP was invented. In the end software development ever suffers from demand for high quality but low effort. But without being sensitive to the danger of what may happen when software grows, bad things will happen.

EDIT

I should perhaps highlight that XML and binary, as long as used by me, is meant not as generic but as specific file format, i.e. "my game's specialized XML file format" and "my game's specialized binary file format".

/EDIT

Yes, I mean that if the ResourceManager object has determined that a specific resource requested by a client is not available from the cache and hence need to be loaded, it delegates the loading to a ResourceLoader object.

How the ResourceLoader class works is another question. It could be an abstract base class, and XMLResourceLoader and BinaryResourceLoader are classes derived by inheriting ResourceLoader. So ResourceLoader provides the common API used by ResourceManager, and XMLResourceLoader and BinaryResourceLoader each one implement the interface, regarding of loading from an XML or binary source, respectively.

2. Should ResourceManager have all the ResourceContainers (which then contain Resource objects)?

[...]

The manager should have all containers that are needed to manage the resource types the manager is responsible for. I used this "weak" sentence because you may find it useful to manage resources for each game level separately, or (which means an inclusive or) you may find it useful to manage resources separated by type.

[...]

In this discussion we use ResourceManager like a director. The manager knows of the loader, cache, ... and mediates between them; that is its job. When going this route, then there is no need to couple e.g. the ResourceLoader to the ResourceContainer just for the purpose to enable the loader to store the resource directly. The ResourceManager, on the other hand, is coupled to the ResourceContainer anyway, because it needs to check whether a resource is already loaded before wasting time by calling a loader.

Notice that the responsibility of ResourceLoader is to load a requested resource. Whether it is meaningful to load it and what happens to the resource after loading is out of the scope of the loader. Restricting the responsibility like so makes it easier to re-use the classes in other contexts. That is one of the points of maintainability.

3. Do you mean that the Sprite class should not have functions to write/read buffers? If so, do you mean there should be a SpriteSerializer class?

Serializing an object means to transform its data into an external representation. In other words, a specific format is used. But the format can change (e.g. XML vs. binary, or version evolution). To abstract this, we have ResourceLoader (and ResourceWriter if saving is supported). So SpriteSerializer would just be another concretization of ResourceWriter.

Notice please that a sprite (as example for any resource) is just a typed data container including some metadata. The animation sub-system alters its world transform, the graphic renderer reads it for the purpose of rendering, the resource loader is able to write data to it, and so on. The sprite itself, as a resource, does almost nothing by itself. If you want to support ResourceLoader / ResourceWriter directly from the resource, then at most something like the Memento pattern should be used.

Some more things to think about:

You have probably noticed that the loader classes so far are designed to abstract the format of the data source, ignoring the type of resource. (BTW: Here the XMLResourceLoader is not meant to understand XML files in general but XML files formatted to represent your game resources, of course.) Real world file formats often represent a single type of resource (see PNG, JPG, WAV, OBJ, ...). In such cases the concrete ResourceLoader (e.g. PngResourceLoader) is a-priorily defined to result in a specific resource type. But in cases of games we often deal with file formats that provide collections of resources (a.k.a. package files or archive files). So, if you want to support single resource loading, you need to support the internal directory that is inherent to such collections. This can be solved by implementing the concrete ResourceLoader to handle the collection stuff and to revert to other concrete ResourceLoader instances as soon as the type of the resource is determined.

Further, we have not examined the dimension of data sources. So far we load resources from files. That is fine in general because files are already abstractions, since they may be mass storage based or socket based. If ResourceLoader should be agnostic of what kind of file it is dealing with, the file has to be opened external to the ResourceLoader class. IMHO even better, any ResourceLoader should deal with an abstraction DataSource anyway.

Edited by haegarr

##### Share on other sites
Diventurer    1631

Nono, I don't store the frames parameter in Resource.

This map is defined in Resource:

std::map m_parameters;

Then each class that derive from Resource, needs to implement an initialise() function which has to at least do one thing to work with the ResourceManager: set the "class" to the class name. Read my other reply again.

Either way; I disagree that it's wasteful, because it's not like you'll have thousands of resources loaded at once. Even then, it's not that bad. We're talking some kilobytes in total.

I'll agree there probably is a better way to do it though.

Ok, replying to this goes somewhat off topic, but this is very timely for me.

So you do store the parameters as name-value pairs like I suggested. It's a map. I guess that the Parameter class stores some union of admissible parameter types? Anyway, if you don't cache the values to some C++ types, each time you request a parameter from a resource you will need to make O(logN) lookup of the map, performing string comparisons. This may be neglible in some simple 2D game with a few resources, but I would not use it for rendering complex 3D scenes with hundreds of resources (with multitude of parameters) accessed each frame.

Storing the names may not take more than a few kilobytes, but usually such degeneracy suggests that the design could be improved.

These parameters aren't really meant to be accessed every frame though. It's mostly just to help build up the object.

I agree there are better ways though.

##### Share on other sites
Diventurer    1631

-snip-

Thanks a lot for your explanation! It really cleared up a lot of confusion.

I'll definitely start applying this in my library. Seems like it'll clean up quite a bit.

Thanks for actually writing such a long post to explain it to me. :)

##### Share on other sites
haegarr    7372

I'll definitely start applying this in my library. Seems like it'll clean up quite a bit.

Fine :) But remember that it is only one way of all the possibilities. Use whatever is meaningful for you.

For example, in my own resource management, the ResourceLoader is not the part that deals with the file data. Also due to other reasons of de-coupling, I have ResourceType and IOFormat, both being base classes. IOFormat represents the format itself, e.g. it has stuff like "how competent are you to interpret this given block of data?". More important to this thread, however, is that it provides 2 abstract inner classes IOFormat::Importer and IOFormat::Exporter as well as 2 abstract factory methods one for the importer and one for the exporter. These both classes are which actually deal with file formatted data. The ResourceLoader itself is more or less just another thing in-between, since it uses a concrete ResourceType as well as a concrete IOFormat and IOFormat::Importer to fulfill its task.

To come to the point, somebody once said "there is no software problem that cannot be solved by another level of indirection, with the exception of having too many indirections". And so you have to decide time and again when your "too many levels" has been reached ...