Jump to content
  • Advertisement
Sign in to follow this  
jmakitalo

Resource manager for open world game

This topic is 1375 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm making an open world (outdoors) cross-platform 3D game with C++ and trying to rework some of the fundamental parts of the code. For loading resources, I have thus far had pretty specialized routines for different type of resources. I would like to unify and improve this part of the code.
 
As I've learned, there are many approached to implementing resource systems and each game has its own requirements. Below I try to list my requirements and issues.
 
TYPES OF RESOURCES
 
The types of resources my game will have:
- texture
- shader
- vertex array object (VAO)
- sound
- skeletal animation
- particle system
 
Those are pretty self-contained resources. But some resources
will depend on others, such as:
- material -> texture,shader
- mesh -> material, VAO
 
I also get the feeling that VAO is a special type of resource in that it is not referenced by name and not declared outside the execution of the game. Perhaps it deserves a dedicated manager.
 
WHEN TO LOAD
 
The world can become quite large, so it would be essential to support loading and discarding resources during gameplay. I think that mostly this applies to textures, as they usually take the most memory. Controlling which resources are loaded and when may not necessarily be the responsibility of resource managers, but they would need no provide an interface for it.
 
I'm thinking of dividing the world into imaginary rectangular sectors. Only resources required by, say, 9 sectors closest to camera need to be in memory. The engine could keep track of this and ask managers to discard/load resources as camera crosses sector boundaries.
 
SMART POINTERS OR HANDLES
 
I've been learning C++11 and am interested in leveraging its functionalities. I've given thought for smart pointers, but their use for resource managers gives mixed feelings. I think that usually resource managers do not take ownership of resources, but only delegate them to users. So one approach would be to store weak pointers in the manager and deliver shared pointers. However, it is not clear whether a resource should be destroyed immediately after it loses its last user. For example, one could define quotas for resource memory use and only after the quota is reached, are some resources destroyed. I guess that handles would prove more flexible for this type of system, but with handles, the manager usually owns the data (not that this is a drawback by itself).
 
Another advantage of handles is that if a resource is requested, it may not be immediately available or may not even be found in some error situation. It would be nice to be able to handle this gracefully, i.e., the handle would temporarily correspond to some dummy resource.
 
RESOURCE HEADERS
 
All resources (except VAO) are defined in text files (XML). Each resource is an XML element with name attribute. The element may have child elements that describe properties of the resource, such as filename, sound volume, animation frame rate, etc. I would think that this data could be read into memory at startup for all resources. A given resource would then have its header in memory and its status would be "not loaded", until the resource is requested to be loaded (a handle is requested).
 
I also want to be able to modify resource parameters in-game. Thus I have written a template class CVar, which is given some type and it can be easily read from XML and can be fetched for an in-game menu system for inspection and modification. A group of CVar objects (CVarGroup) is then given for each resource.
 
I could just store resources of particular type in a vector, which is fully populated at init from the XML file. Then an index to the resource would remain well-defined over the course of the game. A handle would then contain this index, which could point to some dummy data if the resource is not yet loaded.
 
TYPE SAFE STORAGE
 
A major problem I have not figured out yet is how to store the various types of resources in a convenient way. I could make a templated resource manager, that stores a vector of given type. But then I would need to have a separate instance of the manager for each type of resource. Also, as shown above, some resources need to invoke other resource managers, which could then be difficult (loading mesh entails loading material, which then loads textures).
 
On the other hand, I could write a resource manager, which only stores pointers to some resource base class. But in this approach, it seems that dynamic typecasts would have to be performed in hot rendering code. However, this approach could give rise to a really nice interface, such as
 
Handle<Mesh> mesh = manager.get<Mesh>("house");
 
where only a single manager object is used to get all types of resources.
 
A related matter is cache utilization. Is it worth the trouble to try to store the resources contiguously in memory or just as pointers? I would think that, e.g., when rendering meshes in game, it is not likely that mesh data required to draw visible mesh objects would be within a cache line anyway in a complex world.
 
SORTING BY RESOURCES
 
The resource should store some integer that can be use to contruct a sorting key for rendering. For example, meshes should be sorted by shader, material and VAO, so each resource should provide an integer. In the best case, these integer types would be only a few bits long so that they could be combined into one integer type. This should not be a major problem, though. If resources are stored in vector, then the vector index may serve as such integer.
 
MY ATTEMPT
 
I have written some preliminary code, which is based on handles and the approach that each resource has its own manager. Some methods are just there without implementation to sketch it out. I tried to solve the resource manager interdependency by dependency injection, which is illustrated with barebones implementations of material and mesh resources. Please find the code at
 
 
I'm always trying to avoid the singleton pattern so that the managers are objects in some world class.
 
I'm interested in any kind of feedback to improve my approach. Hopefully my goal is not too vaguely stated.

Share this post


Link to post
Share on other sites
Advertisement

You need to create a manager for each type of resource. It can be your compiled shader, shader, model, mesh, render target, etc.

The Single Responsibility of the manager is to avoid loading the resource multiple times. You can also put some kind of logic responsability to the class, but it must be related only with the type of resource you're managing

 

Eg.:

CTexture* ptTexture = CTextureManager::CreateTexture( const CString& _sFilePath ) ->There is a texture? Yes: return it. No: create it.

 

The CTextureManager is the most simple example of a resource manager, but in order to work it must be linked with a "CImageManager" and the "CGraphicsManager" to get all pixel valures and upload the data to GPU.

 

As a good advice you can search on the forums: Game Engine Architeture; after that you can go directly to what you need - basically the  components of the engine - do using the forums.

Edited by Irlan

Share this post


Link to post
Share on other sites


You need to create a manager for each type of resource. It can be your compiled shader, shader, model, mesh, render target, etc.

The Single Responsibility of the manager is to avoid loading the resource multiple times. You can also put some kind of logic responsability to the class, but it must be related only with the type of resource you're managing.

 

I'm not sure I follow. Preventing the loading of resources multiple times is not by itself a sufficient reason to make a separate manager for each type of resource. The resource identifier (hash) can contain the type of resource. The main reason I see for having separate managers is to avoid costly dynamic type casting in hot code.

 

What sort of logic are you talking about? Perhaps that each resource may have totally different loading procedure? In this case the loading functionality could be a virtual function in a resource base class. I'm not sure if this is the way to go, i.e., to let resources load themselves. In my initial code, I let the resource managers load the resource so that the resources themselves have less responsibility.

 


Eg.:

CTexture* ptTexture = CTextureManager::CreateTexture( const CString& _sFilePath ) ->There is a texture? Yes: return it. No: create it.

 

If CTexture is the actual data, I would like to avoid this type of approach. This is pretty close to what I already have in my game engine, although I try to avoid static methods and instead use manager objects, which are instantiated in the engine class.

 

The problem here is that if the manager decides to destroy the texture or move it in memory for some optimization, the user of the resource has a dangling pointer, which will lead to access violation. Also, if using data streaming, the data may not be immediately available. That's why I would prefer handles.

 


The CTextureManager is the most simple example of a resource manager, but in order to work it must be linked with a "CImageManager" and the "CGraphicsManager" to get all pixel valures and upload the data to GPU.

 

Because your CTextureManager depends on other managers, I would consider it "not simple" (if we neglect the actual code for loading various types of resource). I think I will make my texture manager (OpenGL textures) independent of any other manager. Any pixel manipulations can be done with plain old functions and I don't need to store the pixel data in RAM except during loading phase.

 


As a good advice you can search on the forums: Game Engine Architeture; after that you can go directly to what you need - basically the components of the engine - do using the forums.

 

I'm not exactly in the beginning of making a game engine. I have one running, but I want to consider rewriting many parts of it now that I have more clear picture of what I need for my game. I think that in my engine, the two things that require most abstraction are resource managers and entities. Other pieces of the engine are rather self-contained and usualy lead to only a single instantiation (e.g. terrain, special effects, shadowmap system). In my current game engine, I don't have a unified framework for resource managers and entities. I have decided to go with an existing ECS code with the latter, so now I'm focusing on the resource manager problem.

 

I did google around with "game resource/asset manager c++" before this thread. Most discussions are about plain pointer/smart pointer/handle aspect. This is one part of the topic and I'm starting to like the idea of handles, even though I'm currently keen on using smart pointers (maybe without proper judgement). Also the storage of varying types has been discussed and often some boost class, such as boost::any, is suggested. I'm not sure if I want to introduce boost dependency to my project without proper cause. My understanding is that even if using boost::any, dynamic type casting will be necessary.

 

For handles, the article http://scottbilas.com/publications/gem-resmgr/ is cited a lot. It's a good text on handles, but I'm not sure that the sample code on actual resource management is what I'm looking for.

Share this post


Link to post
Share on other sites

 


You need to create a manager for each type of resource. It can be your compiled shader, shader, model, mesh, render target, etc.

The Single Responsibility of the manager is to avoid loading the resource multiple times. You can also put some kind of logic responsability to the class, but it must be related only with the type of resource you're managing.

 

I'm not sure I follow. Preventing the loading of resources multiple times is not by itself a sufficient reason to make a separate manager for each type of resource. The resource identifier (hash) can contain the type of resource. The main reason I see for having separate managers is to avoid costly dynamic type casting in hot code.

 

What sort of logic are you talking about? Perhaps that each resource may have totally different loading procedure? In this case the loading functionality could be a virtual function in a resource base class. I'm not sure if this is the way to go, i.e., to let resources load themselves. In my initial code, I let the resource managers load the resource so that the resources themselves have less responsibility.

 

 

 


Eg.:

CTexture* ptTexture = CTextureManager::CreateTexture( const CString& _sFilePath ) ->There is a texture? Yes: return it. No: create it.

 

If CTexture is the actual data, I would like to avoid this type of approach. This is pretty close to what I already have in my game engine, although I try to avoid static methods and instead use manager objects, which are instantiated in the engine class.

 

The problem here is that if the manager decides to destroy the texture or move it in memory for some optimization, the user of the resource has a dangling pointer, which will lead to access violation. Also, if using data streaming, the data may not be immediately available. That's why I would prefer handles.

 

 

 


The CTextureManager is the most simple example of a resource manager, but in order to work it must be linked with a "CImageManager" and the "CGraphicsManager" to get all pixel valures and upload the data to GPU.

 

Because your CTextureManager depends on other managers, I would consider it "not simple" (if we neglect the actual code for loading various types of resource). I think I will make my texture manager (OpenGL textures) independent of any other manager. Any pixel manipulations can be done with plain old functions and I don't need to store the pixel data in RAM except during loading phase.

 

 

 


As a good advice you can search on the forums: Game Engine Architeture; after that you can go directly to what you need - basically the components of the engine - do using the forums.

 

I'm not exactly in the beginning of making a game engine. I have one running, but I want to consider rewriting many parts of it now that I have more clear picture of what I need for my game. I think that in my engine, the two things that require most abstraction are resource managers and entities. Other pieces of the engine are rather self-contained and usualy lead to only a single instantiation (e.g. terrain, special effects, shadowmap system). In my current game engine, I don't have a unified framework for resource managers and entities. I have decided to go with an existing ECS code with the latter, so now I'm focusing on the resource manager problem.

 

I did google around with "game resource/asset manager c++" before this thread. Most discussions are about plain pointer/smart pointer/handle aspect. This is one part of the topic and I'm starting to like the idea of handles, even though I'm currently keen on using smart pointers (maybe without proper judgement). Also the storage of varying types has been discussed and often some boost class, such as boost::any, is suggested. I'm not sure if I want to introduce boost dependency to my project without proper cause. My understanding is that even if using boost::any, dynamic type casting will be necessary.

 

For handles, the article http://scottbilas.com/publications/gem-resmgr/ is cited a lot. It's a good text on handles, but I'm not sure that the sample code on actual resource management is what I'm looking for.

 

#A I'm just saying that at the very end you'll end with something like what I've said. It is better to start specializing your classes even if your implementation its very basic. Of course, you don't need to have a "CTexture2DManager" if you want; just a "CTextureManager" it's fine if you don't want over engineer. I am saying that it is easy to manage your resources when you have this kind of specific data. Make an std::map < std::string, CTexture* > and you're fine for your first implementation; don't write code more than once; use smart pointers or make an template that deletes all resource pointers of the std::map when necessary.

 

#B You can let resources load themselves because they're the only one that have information of itself. You can search on the forums. Having web acess and don't use for this type of mission it's a waste of time and I think that everyone is tired of answering this specific type of question. Many engines handle resource loading in many ways. You may want to define which way is better for you.

 

#C Most of the questions on the forums are very specific. They won't give "The Asset Manager" for you. What I would recommend is that you do whatever you want and focus on the low-level sub-systems of the engine.

 

Since you're saying that you want simple things, it is impossible to take the right path without having complexity added to the equation.

Edited by Irlan

Share this post


Link to post
Share on other sites

The general case manager I use for a lot of things works well.

 

You just need to add a reference count to each asset.

 

So something along the lines of.

public class AssetManager
{
private:
       Dictionary<Hash,AssetType> loaded_assets = new Dictionary<Hash,AssetType>();
 
public :
       Asset AddAsset(String name)
       {
              if (loaded_assets.Contains(name.Hash))
              {
                     loaded_assets[name.Hash].ReferenceCount++;
              }else{
                    loaded_assets.Add(new AssetType(name));
              }
              return loaded_assets[name.Hash].Asset;
        }
 
        void RemoveAsset(String name)
        {
                loaded_assets[name.Hash].ReferenceCount--;
                if (load_assets[name.Hash].ReferenceCount==0)
                {
                         loaded_assets.Remove(name.Hash);
                }
        }
 
}
Edited by Stainless

Share this post


Link to post
Share on other sites


#A I'm just saying that at the very end you'll end with something like what I've said. It is better to start specializing your classes even if your implementation its very basic. Of course, you don't need to have a "CTexture2DManager" if you want; just a "CTextureManager" it's fine if you don't want over engineer. I am saying that it is easy to manage your resources when you have this kind of specific data. Make an std::map < std::string, CTexture* > and you're fine for your first implementation. Don't write code more than once. Use smart pointers or make an template that deletes all resource pointers of the std::map when necessary.

 

Thanks for you input Irlan.

 

Like I said, I have an implementation already. I have done simple non-abstract resource managers before. Now I wan't to take the next step. I use "std::map < std::string, CTexture* >" type of approach, but have ended up writing similar hard coded approaches to the different managers. Now I'm asking advice for the various aspects of resource managers.

 


#B You can let resources load themselves because they're the only one that have information of itself. You can search on the forums. Having web acess and don't use for this type of mission it's a waste of time and I think that everyone is tired of answering this specific type of question. Many engines handle resource loading in many ways. You may want to define which way is better for you.

 

I would like to have some opinions about this type of aspects. Letting resource load themselves may appear reasonable at first sight, but I have my doubts, because they may not have all the information required to load themselves. For example, if mesh loads itself, it will have to known how to load the materials. I don't find it very natural that resources have access to other managers. This is similar to the question "should mesh objects draw themselves". Nowdays I'm convinced that the answer is no, because they should not even have sufficient information to perform such a task (access to framebuffers, VAO managers, relation to other mesh objects, etc.).

 


#C Most of the questions on the forums are very specific. They won't give "The Asset Manager" for you. What I would recommend is that you do whatever you want and focus on the low-level sub-systems of the engine.

 

I agree. My topic is broad, but I made an effort to specify my requirements. I'm not asking for "the asset manager", but I would like to hear some common pitfalls to be avoided and if there are some approaches that have worked in games similar to mine. Specifically, it would be nice to read about any potential pitfalls in the code I provided and whether it is even going in the right direction, considering the goals I listed.

 


Since you're saying that you want simple things, it is impossible to take the right path without having complexity added to the equation.

 

Well, I didn't exactly say this, but I do wan't to avoid completely over engineering.

Share this post


Link to post
Share on other sites

 

The general case manager I use for a lot of things works well.

 

You just need to add a reference count to each asset.

 

So something along the lines of.

public class AssetManager
{
private:
       Dictionary<Hash,AssetType> loaded_assets = new Dictionary<Hash,AssetType>();
 
public :
       Asset AddAsset(String name)
       {
              if (loaded_assets.Contains(name.Hash))
              {
                     loaded_assets[name.Hash].ReferenceCount++;
              }else{
                    loaded_assets.Add(new AssetType(name));
              }
              return loaded_assets[name.Hash].Asset;
        }
 
        void RemoveAsset(String name)
        {
                loaded_assets[name.Hash].ReferenceCount--;
                if (load_assets[name.Hash].ReferenceCount==0)
                {
                         loaded_assets.Remove(name.Hash);
                }
        }
 
}

 

Thanks!

 

Implementing reference counting is one part of a manager, but perhaps c++11 smart pointers would do a better job if one is only worried about this aspect. I guess that your manager is only supposed to add reference counting on top of dictionary. Some further remarks:

 

- Your code clearly takes the path that there'll be one manager per resource type. That's already a major desicion. I would like to hear more experiences whether this tends to work or not in games that have many types of resources.

 

- Your manager directly returns the asset data, not a handle. Another big desicion, which I think won't work well if the manager decides to move that data in memory or the data is destroyed and a dangling pointer remains.

 

- There seems to be no way to actually load the resource. Or is the idea that after invoking AddAsset(), the asset is loaded? There would have to be some way then to know if it was already loaded.

 

- It does not seem robust that to remove a resource, one has to call a method of the manager. If some game object using a resource is destroyed, it will have to signal the manager that the resource is not used anymore. Injecting manager dependency to game objects does not seem natural.

Share this post


Link to post
Share on other sites

#A You're welcome.

 

#B There are basic two ways of letting this not happening. First: you can have pointers to different dependencies; you need to connect the "CImageManager" to the "CTextureManager". Second: if you're using the static classes approach (not the singleton one) you can just include your header file in the class if you want to make things "simple". The same applies to the "CModelManager". The "CTextureManager" should be connected to it. The Factory Pattern approach it is a good way of creating resources that are inherited. But it is just an example; a good way of saying I'll give  you an ID and filepath and you can return a resource for me.

 

#C I know  what you're saying. Since you know that you're not mixing responsibilities - you have to have that cognitive ability of the classes responsibility - you can do whatever you want to load the resources. But if you're talking about optimizations, it is impossible to answer because we don't know the overall architeture of your engine.

 

#D "[...] depends on other managers, I would consider it "not simple"..."

 

I forgot to Quote. But this is a reply of the Posted Today, 10:25 AM.

Edited by Irlan

Share this post


Link to post
Share on other sites


#B There are basic two ways of letting this not happening. First: you can have pointers to different dependencies; you need to connect the "CImageManager" to the "CTextureManager". Second: if you're using the static classes approach (not the singleton one) you can just include your header file in the class if you want to make things "simple". The same applies to the "CModelManager". The "CTextureManager" should be connected to it. The Factory Pattern approach it is a good way of creating resources that are inherited. But it is just an example; a good way of saying I'll give you an ID and filepath and you can return a resource for me.

 

Connecting the managers together seems more natural (than connecting a type of resource to another type of manager). I did this in my sample code: CMeshManager defined on line 267 stores a pointer to CMaterialManager. This is then used in the load() method.

 


#C I know what you're saying. Since you know that you're not mixing responsibilities - you have to have that cognitive ability of the classes responsibility - you can do whatever you want to load the resources. But if you're talking about optimizations, it is impossible to answer because we don't know the overall architeture of your engine.

 

There are certain aspects of resource manager that need to be fast and some aspects that don't. For example, getting the actual resource data, say texture id, in hot rendering code should be fast. That's why I think that "single manager containing all type of resources" is potentially slow due to required dynamic type casts. One thing that does not need to be that fast is obtaining a handle to the resource by resource name/hash (my sample code line 148), because this is usualy done at load time. In my sample code, I experimented with the idea that at load time, handles to resources are retrieved. At active game time, handles are used to access the actual data. This data should always be usable, but it may not be the actual resource data, if some error occured or the data is still being loaded. This is why I invoked the idea of "dummy" data: each container would have to have one resource that is used in case of error. The engine would have to ensure that such resource is always added before anything else.

Share this post


Link to post
Share on other sites

Some improvements that I have figured out for the sample code:

 

- I got the idea of "dummy" resource while making the code, but didn't really make it solid part of the code. I should make some "const size_t dummyIndex = 0;" to make it more explicit.

- The discard() method on line 62 may not work correctly. If the handle destructor is called after discard(), then the counter of resource at index 0 is decremented. I should also make some "const size_t invalidIndex = std::numeric_limits<std::size_t>::max();" and use that in discard(). I guess that the discard() method would be the main protocol for game objects to communicate that a resource is no longer used. The destructor ensures that this happens if one forgets to call discard() explicitly.

- Perhaps the pointers to resource managers, stored by handles and resources, should be shared_ptr to ensure that if for some reason the manager is destroyed while there are still handles stored, there won't be access violation.

 

On line 315 there is an example of the potential drawback of having multiple managers. Those type of expressions will be repeated. Any ideas how to streamline this?

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!