Designing a Resource Manager Framework

Started by
14 comments, last by nicba 21 years, 2 months ago
Last night I just lost my whole library of loader classes by accident, texture loaders, material loaders, mesh loaders ect. Instead of just sitting down and coding them all up again as they were, I thought I would make things a bit interesting by designing myself a proper resource management framework (not that I actually need such a monster for the small games I write, but it''ll be more fun than just re-doing my ball of mud!). Here''s some of the things I have thought of until now: 1) A couple of different ResourceManagers, which can be assembled into a composite structure (the pattern). So I have a global ResourceManager to retrieve my resources from in-game, but also access to different sub-managers for configuring specific things about certain kind of resources. For example a TextureManager that lets me set the filtering method (mipmapped ect.) on the textures to be loaded. 2) This setup would even allow me to attach different strategies to the different managers for implementing various memory-management/disk swapping algorithms for each kind of resource (by use of the strategy pattern). I would like to separate the management of the resources from the actual loading of them. Also, I would like the framework to be extensible enough to load resources from .res files included in the .exe (screensavers), plain from the file system and from archive files (ex.: .zip). I like my resource loading code to receive just a stream (any kind of stream) and parse that into the requested resource type "transparently." My question is how to accomplish this best. I''m not sure how I ought to divide the responsibility among my classes. In my first solution I have two kinds of classes: ResourceManagers and Resources. The resource managers derive from AbstractResourceManager and the resources derive from AbstractResource. The inheritance hierachy for resources is three levels deep. For example I could have a concrete TGATexture, which derives from an abstract Texture type, which in turn derives from the AbstractResource class. My resource managers should be flexible with regards to supported file formats, so instead of hardcoding them to operate with the concrete resource classes, they only know about the abstract types at compile time. At run-time one should be able to register support for different file formats by passing references to the concrete classes to the managers. Ex:
  
ATextureManger = new(TextureManager);
ATextureManager.registerFileType(TGATexture); ATextureManager.registerFileType(BMPTexture);
....
  
When the game requests a resource from a resource manager, it should attempt to load it by querying all its registered file types classes for one capable of handling it, in a Chian-of-Responsibility like fashion. A TGATexture class would thus know how to handle a ''.tga'' file type and respond to this. By consequence, each concrete resource type should therefore know how to load/unload the resource from memory, using their choosen file format. A TGATexture might therefore have an interface like the following:
  
class TGATexture : Texture
{
  load(stream);  // Load a ''.tga'' file into a Texture structure

  unload();      // Unload the texture (temporarily) from mem.

}
  
Where the stream is opened by a resource manager to that managers respective repository (a file in a specific directory, in a .zip file ect.). There is, however, at least one problem with this approach to resource loading. Tying the the loading of resources directly to subclasses of the abstract resource types, such as with the TGATexture subclass, creates a one-to-one mapping between resource types and files (that is, there is an abstract resource type Texture and one or more concrete implementations that fills Texture with data). The problem arrives with "library" files, as the .mtl files used to store materials for a wavefront .obj model. A .mtl file does not correspond directly to a material, so we can''t have a Material type with a MTLMaterial subclass loading from a .mtl file. The .ntl files contains a collection of materials. Therefore, in order to support these files it would be necessary to define a new resource type, a MaterialLibrary resource. This resource, I feel, would be kind of articficial. The rest of the game/application does not care about material libraries. It cares only about a single material at a time, when its going to render it on a face. Maybe a viable alternative is to have not two but three kinds of classes: ResourceManagers, Resources and Loaders. That is, to split the loading code out in its own class hierachy. There would only be a few kinds of resources: AbstractResource, TextureResource, MaterialResource, MeshResource and AudioResource, for example. And these would all be entirely passive, only storing the data put into them. Instead of registering file types with the resource managers, one would register Loaders. For example there would be a only one TextureResource class, but more loaders: TGATextureLoader, BMPTextureLoader ect. Loaders would be a kind of factory classes, hiding the complexity of instantiating and loading a resource from a file (or a stream, rather). But they would be very thightly coupled to the resource classes storing the textures, materials or whatever. I kind of like the conceptual picture: ResourceMangers in which you registers Loaders for different file format, which in turn instantiates and loads resource for the managers. But then again its another, separate class hierachy. That model would also mean an increase in complexity. I can''t decide which approach I like best.... Any of you done anything like this before? Comments, feedback, tips and ideas are very much welcome. Regards Nicolai
Advertisement
It is absolutely STUPID to have a game engine able to load tons of different formats, like png jpg tga whatever. You want your game to load fast right? Then store all resources ONLY in formats that requires the least amount of conversion to read into your memory structures and classes. This is a very important concept that many programmers forget now in the days of tons of memory and huge cpu speed. Still, games require more and more resources and you don''t want to have your players sit through 5-minute loading times.

For textures, that would be the .dds format, which is really fast to load since (if i''ve understood correctly) it is made for storing textures in a hardware friendly way. Use the directx texture tool to convert your textures to dds, where you have maximum control over the texture format (switch between DXT texture compression and 16-bit and 32-bit to see which looks "good enough" and is smallest).
Well, AP, I''m forced to agree with you there, more or less.

You''re right, it doesn''t make much sense to make a game engine capable of loading a hundred different texture formats. And if that was really necesarry, I could just use DevIL and be done with it.

However, the goal of my efforst isn''t so much to be able to handle many different texture formats. I''ve done alright with just a .tga loader for quite some time. No, what I''m more interested in is creating a framework where I can load and store many different kinds of resources, in as similar a way as posible: Textures, Materials, Meshes, Animations, Sounds and so on.

I would like to be able to do something like the following at the startup of my game:


  ResourceManager.Load(''sometexture.tga'');ResourceManager.Load(''someothertexture.tga'');ResourceManager.Load(''spaceship.obj'');ResourceManager.Load(''explosion.wav'');...  


Or maybe just ResourceManager.LoadAll() to load all resources at a pre-configured path, or ResourceManager.Load(''level1.level'') to load all the level data from a custom level script. Later in the game I would then use a ResourceManager.GetResource() function to retrieve a reference to a specific resource when needed for rendering/playing ect.

The resource managers should be there to ensure that no resource is loaded twice, and that textures, materials and meshes ect. are shared between the in-game entities.

Sure, I could do well enough using DevIL for loading textures, FMOD for sound, some library for models and so on. Accessing these API''s directly instead of going through some ''framework'' would probably be just as easy for my smallish projects.

But then again, as I stated in my first post I''m not really doing this because I *need* such a thing as a resource management framework, right now. I''m doing it because I found it an interesting challenge.
What''s the responsibility of the resource manager - to load the stuff off disk, or just make certain it''s only loaded once?
- The trade-off between price and quality does not exist in Japan. Rather, the idea that high quality brings on cost reduction is widely accepted.-- Tajima & Matsubara
quote:Original post by Anonymous Poster
It is absolutely STUPID to have a game engine able to load tons of different formats, like png jpg tga whatever.


I beg to differ. I intend my game engine to be able to be extended to be a 3d modelling software. So I have a generic 3d engine, which needs to import/export as many different formats as possible.

Nicba, you are almost on the right track with seperating into loader, resource, managers. However, I advise not having a common ResourceManager (fascade pattern, I believe) as animations, meshes, texture are conceptually different items. You should be using a generic resource manager (hint, templates) instead to handle ref counting and such.

quote:
What's the responsibility of the resource manager - to load the stuff off disk, or just make certain it's only loaded once?


It was my intend to make the resource manager(s) responsible for the following three things in particular:

1) Insulate clients from the varying nature of resources by providing a generic interface for loading resources into a common resource pool, and for retrieving them from the resource pool again.

2) Insulate the clients _and_ the loaders from the media upon which the resources are stored (could be either a normal file, in a zip archive, included as a resource in the .exe ect.). The thought was that the resource manager could open a stream to the resource and then hand the stream to the loader code.

3) Make sure that each resource is loaded only once and shared properly between "game entities"

In addition I thought it could be fun to allow resource managers to swap out resources temporarily if necesarry, based on some configurable strategy. But that's not really relevant for my need right now, and not high on my wishlist. It seems also kind of tricky.

quote:
Nicba, you are almost on the right track with seperating into loader, resource, managers. However, I advise not having a common ResourceManager (fascade pattern, I believe) as animations, meshes, texture are conceptually different items.


Do you really think so? Sure, they are different from each other and the RenderSystem, the SoundSystem ect. of the engine would of course know the difference, and know the makeup of "their own" brand of resources.

But seen from the perspecitive of loading them, storing them in memory and retrieving them for use, they still seem to me to be just large memory chunks. Data to be managed. And it would seem logical and useful to have one facade to manage them through.

As for your tip on using templates, I'm afraid I've got very little knowledge about how they're supposed to be used. I've been doing most of my programming in Java and Delphi, which, to my knowledge at least, doesn't have any templates.


[edited by - nicba on February 3, 2003 3:25:39 PM]
quote:Original post by nicba
2) Insulate the clients _and_ the loaders from the media upon which the resources are stored (could be either a normal file, in a zip archive, included as a resource in the .exe ect.). The thought was that the resource manager could open a stream to the resource and then hand the stream to the loader code.


this is were i see a problem. i wouldn''t integrate this functionality into your concept of loaders, resources and
resourcermanagers. it doesnt''t belong there. i am not an native english speaker, but i believe that the words that i need here are: "data access is part of a different problem domain."

perhaps you could say that you will want to separate "data access" from "data handling".

here is an idea:


  // strategy for data accessstruct DataAccessStrategy{    // simply put the raw data into a buffer of any type    // (void*, char*, byte [unsigned char*], your own class)      virtual std::vector<char> GetRawData(const std::string& ResourceName) = 0;        // ... what you want};// example strategy for ZIPclass ZipAccess : public DataAccessStrategy{public:    // set-up your ZipReader    ZipAccess(const std::string& ArchiveName);            // get it from a zip-file    virtual std::vector<char> GetRawData(const std::string& ResourceName);    private:    // your zip-reading classes    ZipReader* mReader;};// your resource interface struct Resource{    // load using the provided strategy    virtual bool Load(DataAccessStrategy& Access);            // etc, like GetSize(), Touch()};// example resource, Image is your Image base classclass TgaResource : public Image{public:    // load the tga format    virtual bool Load(DataAccessStrategy& Access);        // stuff from your image class, eg.    virtual int GetBpp() const;    virtual const char* GetBuffer() const;    private:        // your tga code    TgaHeader* mHeader;    TgaImageData* mImageData;};// manager<template class Resourcetype>class ResourceManager{public:        ResourceType* GetResource(std::string& ResourceName);};  


please note that this is just a very basic idea. it is not supposed to be a good design or even good code.
but it may give you a new perspective.

i hope that i helps at least a little bit.











quote:Original post by nicba
Do you really think so? Sure, they are different from each other and the RenderSystem, the SoundSystem ect. of the engine would of course know the difference, and know the makeup of "their own" brand of resources.


Because in the game engine, you often need to do something special with the graphic resource, sound resource, animation data etc.

If you lump them through 1 fascade, it''s harder to extend. Unless you intend to call your resource using Resource.Load/Play/Unload only, it''s better to seperate them.

I would even put a layer above the resource manager that handles the stream loading, file id management. Your resource manager is has too much responsibilities for a single class, and that''s a very hard to extend design.

quote:
As for your tip on using templates, I''m afraid I''ve got very little knowledge about how they''re supposed to be used. I''ve been doing most of my programming in Java and Delphi, which, to my knowledge at least, doesn''t have any templates.


You r in luck. Wait for an upcoming article of mine.
quote:
this is were i see a problem. i wouldn''t integrate this functionality into your concept of loaders, resources and
resourcermanagers. it doesnt''t belong there. i am not an native english speaker, but i believe that the words that i need here are: "data access is part of a different problem domain."


Yeah, you''re probably right there. And I like your idea about the DataAccessStrategy. Thanks!

quote:
Because in the game engine, you often need to do something special with the graphic resource, sound resource, animation data etc.

If you lump them through 1 fascade, it''s harder to extend. Unless you intend to call your resource using Resource.Load/Play/Unload only, it''s better to seperate them.


Hmmm, yes. But I wasn''t thinking of handling the resources themselves, as such, through the resource managers. Only to "store" them there.

I agree it would be inflexible to impose one facade limiting the interaction with resources to just Resource.Load/Play/Unload. The RenderSystem, for example, would probably want to extract specific groups/submeshes from all the meshes in a scene and sort them according to materials before rendering. That would be impossible if the rendering code was placed in a Resource.Play/Resource.Execute method or similar.

Therefore the resource types should only contain data, I think. Not methods to actually render/play/execute the resource. Initially I would load a resource by calling ResourceManager.load(resourceName). Then at some point I would obtain a reference to the resource by querying the resourcemanager with a statement such as resourceRef = ResourceManager.get(resourceName). And then, if the resource was a mesh, for example, I would hand it over to the RenderSystem for further processing: RenderSystem.render(resourceRef).

quote:
I would even put a layer above the resource manager that handles the stream loading, file id management. Your resource manager is has too much responsibilities for a single class, and that''s a very hard to extend design.


Yes, you''re probably right in this. But "Above" the resource manager? I thought below, or besides (such as the DataAccessStrategy mentioned by the AP) would be more natural. How do mean "above"? Please explain .

quote:
You r in luck. Wait for an upcoming article of mine.


Great! I know I should have looked up templates and generic programming long ago, it''s always healthy to know all the tools at your disposal. But I never got around to it. If I get it served on a silver platter, in a nice article, then I might just get it done...
quote:Original post by nicba
Therefore the resource types should only contain data, I think.


Ideally it should, but you need to take specific hardware/API optimization like vertex buffers in DX.

It''s almost impossible to design a generic container to hold your texture/sound data. For animation and others, it''s possible.

quote:
Yes, you''re probably right in this. But "Above" the resource manager? I thought below, or besides (such as the DataAccessStrategy mentioned by the AP) would be more natural. How do mean "above"? Please explain .


Above meaning a higher layer. The Resource manager only deals with data bytes storage, giving out ids and references to store/access. I wouldn''t use inheritance like AP suggested.

In fact my suggestion is a generic RSM (resource storage manager). It only handles id giving, ref counting, and access. Your loader asks the RSM to return a space, take the id and use the id to access the data from the RSM. So your RSM is sort of a global singeleton (doesn''t have to be a singleton).

For even greater reusability, you parameterise the storage type in the RSM, but all these requires C++ template.

This topic is closed to new replies.

Advertisement