Jump to content
  • Advertisement
Sign in to follow this  
Pilpel

Resource management questions

This topic is 1138 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I know there are many threads regarding this subject but I couldn't find answers to my questions.

I have different kinds of resources. There are OpenGL resources that some of them are too "small" and don't really need a cache system (correct me if I'm wrong), like buffer objects (opengl also pretty much manages them by itself), but others definitely need a cache system, like shader programs and textures. I also have Image objects, mesh geometry and what not.

1. Obviously each resource is loaded separately and works differently, so I'm not sure if I have to create a generic cache class for all of them. I'm talking about something like this:
Cache<Sound*> _soundCache;Cache<Mesh*> _meshCache;Cache<Texture2D*> _textureCache;
Also some people say I should have a "ResourceLoader" class that manages loading from disk. The thing is that some resources like meshes and images are loaded from disk with a different library. For PNG images for example I use libpng, which manages loading .png images by itself. How am I supposed to encapsulate this inside a ResourceLoader class?

2. My second and more important question should be pretty simple but I find it very hard to answer.
Say I have this final GameEngine class. Where do I to put all the resource caches? Do they need to be private data members in GameEngine?
If the answer is yes, then obviously a Mesh resource will need a pointer to the Texture cache, because it loads textures. That means I have to pass pointers through classes all day long.
What options do I have here? What's the most viable approach?

Share this post


Link to post
Share on other sites
Advertisement

I'll try to whip something up here (Pseudocode only cause my c++ skills are near zero)

class ResourceManager
{
    list<Asset> assetCache;
    list<AssetLoader> loaders;

    Asset Load<T>(string path)
    {
        try find asset in assetCache;
            return Asset;

        find correct loader for T in loaders
        {
            return loader.Load(path);
        }
    }
}

interface AssetLoader
{
    Asset Load(path);
}

class FooLoader : AssetLoader
{
    Foo Load(path)
    {
        Do loading stuff here;
        return Foo;
    }
}

class Asset
{
    Type assetType
    string assetName
}

Thats a rough overview of how my system works (WHich is derived from the SharpDX.Toolkit ContentManager. I hope it helps

Share this post


Link to post
Share on other sites

Okay I get the idea. Thanks. This doesn't answer the second question though. Can you answer it?

Edited by Pilpel

Share this post


Link to post
Share on other sites

I can't tell you what is best practice (and there probably is none) but in my engine I have a static class called GameServices in the main executable where I put my ContentManager, Renderer, ScreenManager, etc. into and then I can just do GameServices.GetService<T>().DoStuff();

 

It's probably a crude solution, but I haven't found a better one yet

Share this post


Link to post
Share on other sites
1. Obviously each resource is loaded separately and works differently, so I'm not sure if I have to create a generic cache class for all of them. I'm talking about something like this:

  Cache _soundCache;Cache _meshCache;Cache _textureCache;

Also some people say I should have a "ResourceLoader" class that manages loading from disk. The thing is that some resources like meshes and images are loaded from disk with a different library. For PNG images for example I use libpng, which manages loading .png images by itself. How am I supposed to encapsulate this inside a ResourceLoader class?

In a more sophisticated solution (for a game runtime, not for an editor) resources would not be loaded using generic file formats but by using a format that is more suitable for game runtimes (what means basically that it does not enforce recoding of the data, and even reduce interpretation at this level to a minimum).

 

If you don't have such a file format but several ones then you should also think about several loaders. Lordadmiral Drake has shown this in the pseudocode above. 

 

2. My second and more important question should be pretty simple but I find it very hard to answer.
Say I have this final GameEngine class. Where do I to put all the resource caches? Do they need to be private data members in GameEngine?
If the answer is yes, then obviously a Mesh resource will need a pointer to the Texture cache, because it loads textures. That means I have to pass pointers through classes all day long.
What options do I have here? What's the most viable approach?

IMHO: The approach you describe here does not separate the concerns enough. This probably is the reason of your problem.

 

A mesh resource should not load textures because its purpose is to define a shape. A mesh should not even load itself. A mesh should not even name textures so that a loader can load it. Instead, a (say) model resource should be available that serves as a declaration of which components the model need to be useable (like a bill of materials). When the model is about to be instantiated in the world, the scene management requests it from the content management. The content management requests all listed resources from the resource management. Any addressed resource manager then requests its associated cache and eventually instructs its associated resource loader(s) to load the resource data. The content management returns the instance for the model as soon as all resources (that are marked as mandatory) are available (or, if supported, at least surrogates are available).

 

So, when the game loop runs, it periodically enters a phase where the scene is managed w.r.t. the living instances. Here (somewhere before input and AI control happens) spawning and removal is done. Part of spawning is to make a new instance for the model, and hence requesting all belonging resources. This is the place where the content manager need to be accessed. The content manager hides the fact that each kind of resource has its own resource manager. Further, the resource managers hides the fact whether / how caching and loading are done.

 

BTW: I use the term "manager" with the meaning of a front-end that decides and delegates the actual tasks to attached helper objects. Well, that's the way managers work, isn't it? ;)

Edited by haegarr

Share this post


Link to post
Share on other sites

I have different kinds of resources. There are OpenGL resources that some of them are too "small" and don't really need a cache system (correct me if I'm wrong), like buffer objects (opengl also pretty much manages them by itself), but others definitely need a cache system, like shader programs and textures. I also have Image objects, mesh geometry and what not.

You only really need a cache for files that are loaded from disk (or are generated using expensive procedural generation algorithms). You definitely don't want every single texture/buffer to go through a caching system... but model/texture assets loaded from disk should be cached to avoid double-loading them.

I use a file loader (produces blobs of untyped bytes), factories (templated -- convert blobs of bytes into usable assets of different types), and an simple asset cache (inheritance/virtual -- returns a base asset class that can be cast to the correct type).
 

Also some people say I should have a "ResourceLoader" class that manages loading from disk.

I would make this same recommendation. Lots of parts of an engine need to read data from disk -- so write a great implementation of it once. Don't rewrite file loading routines in every asset factory...
 

For PNG images for example I use libpng, which manages loading .png images by itself. How am I supposed to encapsulate this inside a ResourceLoader class?

Usually these libraries also have a "load from memory" API -- that lets you do the file loading yourself (which is good: you can do it asynchronously), and then pass the loaded blob of bytes to the appropriate factory/library for conversion into a usable asset.
 

obviously a Mesh resource will need a pointer to the Texture cache, because it loads textures. That means I have to pass pointers through classes all day long.

Yes, a MeshFactory (turns raw bytes into a mesh asset), needs a TextureFactory. Passing pointers/references is not that painful as long as your design is well refined.

e.g. this does not seem complex to me...

int main()
{
  BlobLoader     loader("./data/");
  OsWindow       window(OsWindow::Fullscreen);
  GpuDevice      gpu( window );
  TextureFactory factTexture( gpu );
  MeshFactory    factMesh( gpu, factTexture );
  AssetCache     cache;

  Mesh* mesh = cache.Load( "test.mesh", factMesh, loader ); // check the cache, if not found, load the file using loader and parse it using the mesh factory.
  //if the mesh then requires textures, it can do the same thing using it's texture factory member which we passed to it.
}
Edited by Hodgman

Share this post


Link to post
Share on other sites

It will make your life a lot simpler if you have a base file loader that doesn't care about what asset type it is loading. (Hodgman touched on this above).

 

The reason is that on most operating systems you have a finite number of file handles available. So if you have code flying around on different cores, kicking off async disk reads, it is possible to run out of file handles and crash the game.

 

You can also do really nice stuff like load in only the lowest mipmap of a texture when running on a main thread, then create background tasks to load the rest of the mipmaps.

 

This allows your game to load much quicker at the expense of assets looking "a bit shit" until the rest of the mipmaps are loaded.

 

Resource management can be as complex or as simple as you want. It depends on the game design a lot as well as the platform.

 

Let's look at a few cases.

 

1) FIxed asset size.

 

     When I wrote a couple of XBOX 360 indie games, you have 512 meg of assets space. That was it. It made no sense to have a complicated asset management scheme in this case. Just load everything at the start of the game.

 

2) Level based design. Tight control on assets used in the level

 

     Say you have a racing game or something like that where you have a fixed gameplay area. You could set hard limits on how much ram is used for the assets. Again in this case it makes no sense to have a complicated resource manager. Have a single threaded resource loader and load all the assets at the start of the level.

 

3) Level based design, big levels

 

     In this case you probably won't have enough ram available to just load everything and you have to get creative. What I tend to do is split the level into sections and scan the assets breaking them down into two groups. One group contains the assets that are needed everywhere in the level. This group is loaded first and persists through the level. The rest of the assets are broken down by section and have to be loaded on background threads as they become required.

 

4) Open world

   

    Stream everything.

 

 

You need to look at your game design and make a decision about how far you need to go.

Share this post


Link to post
Share on other sites
You can use a common base class for all kinds of resources, and store them in the same cache. Then, for instance, a mesh would ask for a resource named texture.png, and would expect that the resource it received is actually a texture (which can be checked for with a dynamic_cast, or simply casted to it directly if you 'trust' your resources). Then you can have various kinds of resource loaders, which also inherit from a common interface that can parse any given file and returns either a resource object or a null. You then register the loaders with the manager, and whenever you ask for a new resource it either goes through all the loaders to find one that works, or does some magic with the file extension/signature.

Share this post


Link to post
Share on other sites

Thank you all. Here's what I did:

I have this abstract, generic ResourceLoader class

template<class T>
class ResourceLoader
{
public:
	virtual T* load(const char *file) = 0;

private:
};

And now when I want to create a new loader class (for an Image for example), I inherit from ResourceLoader:

class ImageLoader : public ResourceLoader<Image>
{
public:
Image* load(const char *file)
    {
    Image *image = new Image;


    if(image->load(file)) {
        image->read();
        return image;
    }

    return nullptr;
}
private:


};

Then I have this ResourceCache class template, that uses a loader class.

template<class T>
class ResourceCache
{
public:
	ResourceCache(ResourceLoader<T> *resourceLoader) : _resourceLoader(resourceLoader)
	{}
	~ResourceCache();

	T* loadObject(const char *file);
	bool removeObject(const char *file);
	void removeAllObjects();

	unsigned int getNumObjects() const;
	T* getObject(const char *file);

private:
	typedef std::shared_ptr<T> t_ptr;

	ResourceLoader<T> *_resourceLoader;
	std::unordered_map<std::string, t_ptr> _objects; //file, shared ptr of object
};

And finally, here's how it's gonna be used

ImageLoader imageLoader;
ResourceCache<Image> cache(&imageLoader);

const char *file = "C:\\Users\\User\\desktop\\dog.png";
cache.loadObject(file);
Image *myImage = cache.getObject(file);

Any tips?

 

BTW: I don't plan on doing streaming (for now).

Edited by Pilpel

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!