• Advertisement
Sign in to follow this  

Resource Management

This topic is 1048 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

So I want to code a resource management system for my (OpenGL) rendering engine.

I have many different kind of resources: images, opengl textures, vbo's, ebo's, vao's, rbo's, shaders, shader programs. (did I miss something?)

 

What are the coding options for building this kind of manager? I tried looking at other rendering engines' codes but they're hard to understand.

The obvious answer for resources should be something like the following code, while other resources (like shaders and images) will have their own Manager class, and not a general one.

//abstract class
class Object //I called this Object but could also be named Resource
{
public:
	Object() : _id(-1) {};

	virtual void create() = 0;
	virtual void bind() = 0;//doesn't make sense for resources like "image"
	virtual void free() = 0;

	int id() const { return _id; }

protected:
	int _id;
};

class VBO : public Object
{
public:
	VBO() {}
	void create() {};
	void bind() {};
	void free() {};
};

class Manager
{
public:
	Manager() {}
	void add(Object *obj) {
		_map.insert( std::pair<int, Object*>(obj->id(), obj) );
	}
	void free(Object *obj) {
		_map.erase(obj->id());
		obj->free();
	}

	size_t size() const { return _map.size(); }

private:
	std::map<int, Object*> _map;
	Object *_obj;
};

Is this a good approach? Are there any better approaches for resource management systems?

Share this post


Link to post
Share on other sites
Advertisement

I use this approach (which make things while obeing the SRP):

 

For each type of resource I create a manager. The shader manager, texture manager, render target manager (to manage render target switch and shared render targets).

 

Vertex and index buffers they don't need a manager so I just have a model manager since gets into a model class...in the easiest way the managers only avoid to load the resource more than once; if you need to work with some kind of related funcionality there is no problem as long it is related to the resource.

 

A lot of people here handle it in different way; not trying to abstract everything into a single abstract class resource manager it is a good thing also. Resources manage itselves their life-times. Destruction is inside the object destructor and construction can be a function or if you use RAII for resources do it in the way you think is the best for you; I personally handle loading into functions and destruction into a function called Destroy/Reset that gets called on the object's destructor.

Share this post


Link to post
Share on other sites

"Manager" is a term that is so nonspecific as to lead to all manner of functionality being lumped together in one class. Before you start designing a resource system, it's important to decide what it is actually going to do. Is it going to load resources? Is it going to be a cache for resources? Is it going to handle automatically disposing of resources?

 

What you have sketched out above literally does none of these things, which makes it, to my mind, not terribly useful...

Share this post


Link to post
Share on other sites
If you try to abstract a manager you will get back to a specialized class no matter what you do. Resource management needs to be fast as well any other module in the engine, and abstracting you're forced to add complexity in your scheme at some point. A lot of engines out there has some kind of asset manager, but the class is still specialized; each resource does not extend an asset.

What do you need? Do you need to avoid loading multiple resources? Do you want to manage allocated memory? Want the user to easily interact with a resource?

Your implementation it is not wrong; each resource has its class; you only need to define what actually you want with them.

Once the resource it is created and stored somewhere there is always a upper class that manages its life-time being high-level or not.

I would recomend searching for the topic Game Engine Architecture because there arw topics that clarifies your questions in a more advanced way;all you will hear here is problably discussed somewhere; you can take this shortcut only if you want because information is always valuable here no matter if is permuted.

Share this post


Link to post
Share on other sites

It does load resources to a std::map and it is a cache for them as well. What are you seeing that I'm not?

It's not loading anything, because you are requiring the caller to provide an already loaded object to insert into the map, and it's not caching anything, because it provides no interface to lookup items which are already in the map.

Share this post


Link to post
Share on other sites
I have many different kind of resources: images, opengl textures, vbo's, ebo's, vao's, rbo's, shaders, shader programs. (did I miss something?)

I'd say yes.

 

A couple of the things you mentioned require no active management at all. Shaders and shader programs, I very much doubt you'll be loading them on the fly or ever unloading them while a game is running, and the same can be said about RBOs, FBOs and texture attachments for FBOs, these most of the time occupy a very specific role in the pipeline (say g buffer, post prossess, light accumulation buffer, etc), so passing them down to a catch all generic resource manager seems a bit of overkill to me, you'd be architecturing for situations that wont happen. At most you'll want a few functions so you can reallocate FBO/RBO/texture attachment if the resolution is changed, they're ad hoc though, no need to create a whole system to handle them. My stance on these things is always: Code for what you will need, not for what you *might* need.

 

Now the resources that you do want to manage dynamically are vertex buffers and textures. Those will get loaded and unloaded on the fly.

 

As to how to design that resource manager, I don't have much to add. I haven't implemented a fully working resource management scheme. So far the thing I have clear is that file loading should be abstracted. So while the "resource manager" provides common APIs to request for resources, the loaders themselves are created separately and registered into the manager. If the manager lacks a loader to load an specific resource, it will let you know (aka, NPE as soon as you use the result biggrin.png ). Also that caching schemes should be abstracted too, if you want a resource to be cached, you wrap the loader into a cache, and pass it to the resource manager, the manager wont know its using a cache (because it doesn't needs to).

Edited by TheChubu

Share this post


Link to post
Share on other sites


If you try to abstract a manager you will get back to a specialized class no matter what you do. Resource management needs to be fast as well any other module in the engine, and abstracting you're forced to add complexity in your scheme at some point. A lot of engines out there has some kind of asset manager, but the class is still specialized; each resource does not extend an asset.

 

I nearly started reading it wrong the first time, but this is important. You really do not want to have a base resource type that is required to be inherited from. This means there's tight coupling between resources and manager, and also that you cannot use your manager, then, with any existing class/struct or with any primitive type without first writing a wrapper class for it which inherits the base type. This leads to a lot of unnecessary boilerplate -- better would be for resources and their manager to be loosely coupled, then you can readily place any existing type into a properly-specialized manager, or even primitive types. There's no reason your manager should not readily accept, say, std::string, and there's no reason it should be precluded from accepting, say, int or bool types or C-style file handles (which, IIRC, are just integer identifiers), either. Irlan is right, also, about specialization at some point -- Its my experience also that you want separate managers for separate kinds of things (textures, meshes), rather than one uber-manager that holds all resources. You can (and IMO, should) however allow for a manager to manage an abstract type -- eg. one texture manager instead of texture2Dmanger+texture3Dmanager, etc.

 

Also, many people reach to singleton pattern for such managers, but I think this is wrong-headed. Do not allow your design to assume that only one manager of a single resource type (abstract or concrete) exists -- If you design such that many can co-exist, you can choose to use only one and get all the same traits of a system which only allows for one, but you cannot choose to create many if your design enforces only one.

 

Also, everything people have said about the specificity of "Manager" is correct, and I do not think you've clearly defined what your goals are -- Usually, a resource manager implies either a common-point-of-access, resource deduplication, on-demand resource retirement/restoration (for e.g. streaming), pooled allocation, or some combination of these things. It is my experience that each of these properties is best treated as a separate responsibility, and so, is encapsulated into its own class, and higher-level functionality achieved by orchestrating these classes together, either enforced through a higher-level class, or by policy.

Share this post


Link to post
Share on other sites
Ravyne, yeah that was my point.

The OP may not want to ask why singletons are bad because thats a old long history that was started since Dracula was born. Just avoid as much as possible and if you use it go with another type of design such static functions to handle the life-time of the objects; always trying to get a instance based architecture is a good practice :-).

Share this post


Link to post
Share on other sites

Also, many people reach to singleton pattern for such managers, but I think this is wrong-headed. Do not allow your design to assume that only one manager of a single resource type (abstract or concrete) exists -- If you design such that many can co-exist, you can choose to use only one and get all the same traits of a system which only allows for one, but you cannot choose to create many if your design enforces only one.

 

 

So do you claim that singleton patterns are bad habit? I've seen them used many times while viewing engines' source codes.

Edited by Pilpel

Share this post


Link to post
Share on other sites


So do you claim that singleton patterns are bad habit?

For the most part, yes. Though as with any rule, there may be specific exceptions.

 


I've seen them used many times while viewing engines' source codes.

Open-source engines are not necessarily a good place to look for examples of correct software design.

Share this post


Link to post
Share on other sites


So do you claim that singleton patterns are bad habit? I've seen them used many times while viewing engines' source codes.
That's bad, and they should feel bad.

Share this post


Link to post
Share on other sites

In an effort to steer sharply away from a Singleton flame war...

 

 

I also find "manager" too generic.

 

I prefer a combination of a loader, a cache, a proxy that implements the full resource interface, and a store. Each has a clear responsibility. You request an object from the store and immediately receive a proxy. If the object is already in the cache the proxy is filled in with the live value, otherwise the proxy gets a well-established placeholder. In the latter case, the loader pulls the object from the disk/network/other and loads it into the cache, then notifies the user of the proxy. When the engine decides the resource is no longer needed it can remove the object from the cache and update the proxy back to the placeholder data.

 

Edit: There are different classes for each class of resource because they really are different. A story script or game object script can coordinate with the scripting engine. A graphics texture store needs to coordinate with images kept on a graphics card, an audio clip store needs to work with items kept in wwise or another audio system. An animation store needs to interface with the animation system. And so on. While there are many things in common, the details are unique to each.

 

Edit again: Also this system works well with another frequently mentioned aspect of game development; it easily supports hot reloading of resources. The engine can use file system monitoring to detect that a model or texture or audio clip or animation or whatever else has changed in the file tree, then load and swap the modified resource.

Edited by frob
So many reasons why the pattern is so common...

Share this post


Link to post
Share on other sites

Also, many people reach to singleton pattern for such managers, but I think this is wrong-headed. Do not allow your design to assume that only one manager of a single resource type (abstract or concrete) exists -- If you design such that many can co-exist, you can choose to use only one and get all the same traits of a system which only allows for one, but you cannot choose to create many if your design enforces only one.


So do you claim that singleton patterns are bad habit? I've seen them used many times while viewing engines' source codes.
It doesn't matter how you call it; as long its functionality it only one or related to the class itself...

A singleton can be replaced most of the time with a instance based class; I would like to re-iterate: every singleton architecture can be replaced with a instance based one, most of the programmers don't change from a global class architecture to a instance one because sometimes it adds some complexity to your current scheme; the number of classes to wrap the existent architecture can increase if not designed very well.

Calling a manager can be a bad thing, because leads the programmer to think abstractly too much instead of first re-thinking in what needs to be implemented in a specialized class.

A lot of engines uses the manager name and the singleton pattern, but today what matters is do the right thing with code as well many other fields in game development.

I don't have experience in a game company yet, but I believe that with the number of employers in a company today, things should be organized in the best way possible (mainetenable; actually maybe doesn't need to be experienced to note that).

Also, remember that you should organize your engine using modules; thats the only way to NOT do it wrong (at least 60%). If you want an simple example using a intermediary code base you can go into my blog above this quote and search there. Using Visual Studio that can be done using static libraries and solutions to separate the modules responsabilites and it is for sure the best thing to do for a begginner (I'm not saying you're a beginner, just commenting).

Once you have a module make sure to work inside its black box, calling X manager whenever you want without having to make libraries that doesn't depend of another library, also it helps to get most of the things instance based.

Share this post


Link to post
Share on other sites

 


I've seen them used many times while viewing engines' source codes.

Open-source engines are not necessarily a good place to look for examples of correct software design.

 

 

Then can you link a good article or book about this subject? (besides Game Engine Architecture, which I have already read)

Thanks for the comments but that was too much theory lol. I don't know what to begin with...

Edited by Pilpel

Share this post


Link to post
Share on other sites


I have many different kind of resources: images, opengl textures, vbo's, ebo's, vao's, rbo's, shaders, shader programs. (did I miss something?)

Please notice that different kinds of resources requires different handling as well. The usually meant kind is that of assets like meshes, animation data, sound clips, texture images, scripts, and so on. This stuff is designed, resides on mass storage and need to be loaded into memory. On the other hand, texture objects, VBOs, IBOs, RBOs, and such are resources in the sense of production means. They are not loaded from mass storage but requested from the graphics API (OpenGL in your case). Furthermore, when thinking of VBOs and IBOs and similar buffer objects, the memory behind it may also need to be seen as a resource to be managed, for example if you want to batch meshes. Similarly there may be a management of texture areas in the sense of atlases or levels in a texture array object. And at the end of the spectrum there are resources like texture units that are available in limited amount, depending on the running platform, and hence may need to be managed, too.

 

Where to start ... well, that is a good question. In fact you cannot render anything if not both assets and production resources are available. What do you have already? Is loading of assets already implemented? Is a basic renderer already implemented?

Share this post


Link to post
Share on other sites
 

What do you have already? Is loading of assets already implemented?

 

Yes. I encapsulated Assimp in classes. However, the Mesh class I wrote, for example, uses opengl commands, which I think isn't correct design. (this doesn't obey the Single responsibility Principle)
class Mesh
{
public:
	Mesh(aiMesh *mesh) : _mesh(mesh)
	{
		_hasNormals = _mesh->HasNormals();
		_hasTextureCoords = (_mesh->mTextureCoords[0] != 0);

		for(unsigned int i = 0; i < _mesh->mNumVertices; i++)
			_vertices.push_back(glm::vec3(_mesh->mVertices[i].x, _mesh->mVertices[i].y, _mesh->mVertices[i].z));

		if(_hasNormals)
			for(unsigned int i = 0; i < _mesh->mNumVertices; i++)
				_normals.push_back(glm::vec3(_mesh->mNormals[i].x, _mesh->mNormals[i].y, _mesh->mNormals[i].z));

		if(_hasTextureCoords)
			for(unsigned int i = 0; i < _mesh->mNumVertices; i++)
				_textureCoords.push_back(glm::vec2(_mesh->mTextureCoords[0][i].x, _mesh->mTextureCoords[0][i].y));

		for(unsigned int i = 0; i < _mesh->mNumFaces; i++)
		{
			aiFace face = _mesh->mFaces[i];
			for(unsigned int j = 0; j < face.mNumIndices; j++)
				_indices.push_back(face.mIndices[j]);
		}

		glGenVertexArrays(1, &_vao);
		glGenBuffers(1, &_vbo);
		glGenBuffers(1, &_ebo);

		glBindVertexArray(_vao);

		glBindBuffer(GL_ARRAY_BUFFER, _vbo);

		//vertices
		glBufferData(GL_ARRAY_BUFFER, sizeof(glm::vec3) * (_vertices.size() + _normals.size()) + sizeof(glm::vec2) * _textureCoords.size(), (const GLvoid*) _vertices.data(), GL_STATIC_DRAW);
		glVertexAttribPointer(0, 3, GL_FLOAT, 0, 0, 0);
		glEnableVertexAttribArray(0);
		if(_hasNormals) { //normals
			glBufferSubData(GL_ARRAY_BUFFER, sizeof(glm::vec3) * _vertices.size(), sizeof(glm::vec3) * _normals.size(), (const GLvoid*) _normals.data());
			glVertexAttribPointer(1, 3, GL_FLOAT, 0, 0, (const GLvoid*)(sizeof(glm::vec3) * _vertices.size()));
			glEnableVertexAttribArray(1);
		}
		if(_hasTextureCoords) { //uvs
			glBufferSubData(GL_ARRAY_BUFFER, sizeof(glm::vec3) * (_vertices.size() + _normals.size()), sizeof(glm::vec2) * _textureCoords.size(), (const GLvoid*) _textureCoords.data());
			glVertexAttribPointer(2, 3, GL_FLOAT, 0, 0, (const GLvoid*)(sizeof(glm::vec3) * (_vertices.size() + _normals.size())));
			glEnableVertexAttribArray(2);
		}

		glBindBuffer(GL_ARRAY_BUFFER, 0);

		glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, _ebo);
		glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(unsigned int) * _indices.size(), (const GLvoid*) _indices.data(), GL_STATIC_DRAW);
		glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, 0);

		glBindVertexArray(0);
	}

This is bad. First I think I should encapsulate vertex buffers in a class called VBO or something. Accessing direct opengl commands like that is ugly.

Secondly I think that I should not call opengl commands in this class at all. It's only meaning should be to store mesh data (again, the Single responsibility Principle).

I should access this data in a higher level class and then mix it with opengl functions (can anyone approve this approach?). Anyway, like I said, I don't know where to begin. I don't want to write code that will turn out to be bad. That's why I tried viewing other engines' source codes, but they are very hard to understand (for me at least..).

 

Is a basic renderer already implemented?

 

Nope. My intention was to build a resource management system before implementing the renderer, that should rely on it.

 

 

I really want to start coding but I'm stuck too much time on the design. This is really depressing =[

Edited by Pilpel

Share this post


Link to post
Share on other sites

Nope. My intention was to build a resource management system before implementing the renderer, that should rely on it.
 
I really want to start coding but I'm stuck too much time on the design. This is really depressing =[

You can't design systems like these in isolation (at least not until you've built one 2-3x before).

The best case here is that you finish building the resource system, start on the renderer, and then have to rebuild it all. Instead, start building the renderer right away, and you'll find it places explicit demands on the design of your resource system. Edited by swiftcoder

Share this post


Link to post
Share on other sites

Is a basic renderer already implemented?


Nope. My intention was to build a resource management system before implementing the renderer, that should rely on it.
 
 
I really want to start coding but I'm stuck too much time on the design. This is really depressing =[


The most important thing you can do right now is to start coding on your current design.

It won't be "perfect", and you'll almost certainly have to rewrite large portions as you find your design needs to change to match "real world" demands, but that's part of the process. But you won't be able to find the holes or flaws in your design until you're actually working with it.

Design paralysis kills a ton of games (and developers) all the time - don't let it happen to you smile.png

Share this post


Link to post
Share on other sites

As said by previous posters: Start with the mesh class you have, and implement a simple renderer using it. This is the only way to constantly check that things are working. Then begin to refactor your code.

Share this post


Link to post
Share on other sites

The service finder is better than the singleton. The well-known instance (a pointer a static global pointer initialized during program startup that points to the current version) is better than a singleton. Even the global static variable that delays application startup is better than the singleton in nearly every case.

 

In all these cases you can still have an instance that everybody uses, you can still have your hidden background dependency that no code acknowledges. But you can do it without the singleton.

 

Just because you only have one right now does not mean you should prohibit the option to only have one ever.

 

It seems the common example people give of a "good" singleton is a logging system. But even there, what if I am building and debugging a system and want my own private logger that points to a different logging destination for my own debugging purposes? If it is a singleton, I am forbidden from it.

 

Having a single instance of something is not a problem.  There are many objects out there where you only ever need a single instance.  However, that is radically different from the Singleton. The Singleton means you can only have one instance ever, and are forbidden from creating another. A Singleton means you are forbidden from ever creating more than a single instance; it follows that you are also forbidden from copying it, from cloning it, from abstracting it, from subclassing it, from testing it (you cannot use mock objects), from parallelizing it, and so on. 

 

Singleton != happening to only have a single instance.  Singleton means much stricter requirements.

 

 

 

 

There are several development issues that need to be overcome, they can be addressed without the Singleton problem.

 

Shared state is a difficult problem. There are situations where multiple systems need to know about details from other systems, sharing their state. This should be minimized by using accessors and mutators rather than directly modifying values, by using event listeners that get notified when values are changed so they can respond and update, and by other methods. This is a case where you tend to need only a single instance that maintains the state, but that is very different from a Singleton. Remember that happening to have only a single instance != Singleton. 

 

It is common to have a collection of well-known objects. For example, you may have a structure configured at the start of the game containing pointers to the simulation, to the renderer, to resource stores, to game state, and to the application logger. These should still not be implemented as a singleton where there can only be one instance ever, but instead as a central location for discovering the shared information. 

 

It is common for dependencies to be changed at runtime. You might replace entire systems like dropping in a new network controller when you connect or disconnect from servers. Some beginners think this means you need a Singleton class to guarantee that only one item is active. Instead you can use a well-known instance that serves as a redirector to the currently active instance. 

 

 

 

Trying to stay on topic, an example could be a global instance of a structure something like this:

 

struct GloballySharedObjects {

  Simulator* simulator = NULL;

  Renderer* renderer = NULL;

  NetworkCore* network = NULL;

  MeshStore* meshStore = NULL;

  TextureStore* texStore = NULL;

  AudioStore* audioStore = NULL;

  MusicStore* musicStore = NULL;

  ILogger* log = NULL;

  ...

};

 

 

Elsewhere:

 

// Global object so we can share state. The hidden dependencies should be avoided if possible, but still better than singleton.

GloballySharedObjects gs; 

 

 

Then you can build all your parts for resource management: A store, a cache, a loader, and proxy objects. They can be swapped out as needed, all without a singleton.

Share this post


Link to post
Share on other sites

I really want to start coding but I'm stuck too much time on the design. This is really depressing =[

If there is one lesson that I've learned with coding it is that taking this inbuilt sense of "I have to do everything perfectly" that I have and throwing it out the window is the path to greatness.

Coding is one of those things where you more or less cannot design a useful system without having firsthand experience with it. That applies to any coding topic really. Sometimes the best thing to do is just jump in the deep end and fix problems when you run into them.

 

If anyone here started coding a game and were say, asked to code an input handling system and they just threw together some masterpiece of technology I would love to meet them, I would hazard a guess most of us made really pathetic attempts at new specializations of coding even if they had a ton of prior experience in the language, some stuff just takes revision to make pretty.

 

Looking at code and thinking that will teach you and you can just mimic their style also doesn't work, coding isn't like watching someone cut wood with a saw, it has thousands of levels of depth that only time and attempts and coding the thing yourself will really cement into your mind.

Edited by Satharis

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement