Graphics engine structure + resource manager questions
Author
Ok, finally I have started to make a plan on what to do to get my engine up and running. Before, I was making components and not dealing with the bigger issue. So now I have some questions with the bigger issue.
I am using direct x, if that matters.
1) The resource manager holds all resources, textures, ect so they are not loaded twice. Should the resource manager handle any loading, with some exceptions? I am in the middle of moving my terrain engine into a lib, and obviously the raw height data file does not have to be handled with the manager, but should the manager take care of all vertex and index buffer creation and destruction? In that case, what should the resource manager handle? I would assume so, but would like a consensus to see what other people do. I have not dealt really with meshes, so I don’t know how the meshes will fit in here but I am assuming that they will.
2) After putting my thoughts on paper, the graphics engine is made out of a few components, the resource manager, terrain, sky rendering, meshes, particle systems, and water. I cannot think of anymore. There will be a scene manager, which will take care of partitioning, and instancing out game objects (going for the component scene management). Now, the scene manager will take care of the rendering messages, but where should I actually render? The simple case would be a renderer class, but will it have to compute data from the scene manager and the resource manager to put the object together, or is this overly complicated?
As I said, I have not dealt with animated meshes so I don’t know if they will have an adverse effect on my plan.
3) What other graphics engine components should there be? (sound, scripting, input, physics and scene management will not be tied to the graphics engine, however the scene management will require some of the components, such as the resource manager.)
4) Does my plan seem good? Or is it overly complicated and I don’t know what I am talking about or doing?
Also: would the best method to check and see if a resource is loaded be to keep a matching array that stores the filename? then if a request for a texture is send, it will search the texture storage array, if it is founf, give the address to the matching element in the texture array, otherwise, load a new one and add the texture and filename to the respective arrays? Or would it be in an array of structs, which hold filename, and the texture data in that? Or I am assuming there is a more conventional way?
[Edited by - Valor Knight on March 29, 2006 11:11:38 PM]
We need to step back and think in more abstract terms. Stop and think what a resource manager is should be in charge of. "Managing resources," you say. How so? perhaps:
- retrieves resources when requested
- makes sure you don't load the same things twice
- loads resources
Notice we are talking about "resources". Not meshes, sound fx, graphics, or any other thing. This is all rather abstract at this point. Ideally, you want to encapsulate as much of the specifics as possible. For instance, if you want to add sound to your engine, instead of kludging together another part of the resource manager and building a monolithic mess, make your actual resource manager very abstract. Put all the loading/dumping code for specific resources into separate classes. That way you can add new resource types easily (or you can change what libraries you use for those resources without recompiling the actual manager)
So the manager, theoretically, should just be able to say:
ResourceManger::LoadResource( name ) {
appropriate_module.LoadThing( name );
}
All the resource-specific code could go in "appropriate_module".
NOTE: these are just polite and helpfull suggestions. There are plenty of ways to deal with this. YMMV, and all that.
For my manager, i keep names of all my resources and do bookkeeping. I force clients to check in and out resources using their own name. Like this:
Mgr->GetGraphic( "graphic_name", "my_name");
Now, the manager knows two things: 1) that the resource has been checked out exactly once, and 2) that the resource has been checked out by a specific client.
If i run the program, i can at any time get a diagnostic report from the manager that simply lists who has what checked out and how many times.
If all clients check resources in, the resource is dumped and memory freed. Yeah!
- retrieves resources when requested
- makes sure you don't load the same things twice
- loads resources
Notice we are talking about "resources". Not meshes, sound fx, graphics, or any other thing. This is all rather abstract at this point. Ideally, you want to encapsulate as much of the specifics as possible. For instance, if you want to add sound to your engine, instead of kludging together another part of the resource manager and building a monolithic mess, make your actual resource manager very abstract. Put all the loading/dumping code for specific resources into separate classes. That way you can add new resource types easily (or you can change what libraries you use for those resources without recompiling the actual manager)
So the manager, theoretically, should just be able to say:
ResourceManger::LoadResource( name ) {
appropriate_module.LoadThing( name );
}
All the resource-specific code could go in "appropriate_module".
NOTE: these are just polite and helpfull suggestions. There are plenty of ways to deal with this. YMMV, and all that.
For my manager, i keep names of all my resources and do bookkeeping. I force clients to check in and out resources using their own name. Like this:
Mgr->GetGraphic( "graphic_name", "my_name");
Now, the manager knows two things: 1) that the resource has been checked out exactly once, and 2) that the resource has been checked out by a specific client.
If i run the program, i can at any time get a diagnostic report from the manager that simply lists who has what checked out and how many times.
If all clients check resources in, the resource is dumped and memory freed. Yeah!
Author
Wow, thanks for the great reply.
Yea, after I started looking at Ogre's resource manager, I figured it must have to be abstract. But this brings up new questions.
I think I am starting to usderstand the more I looked at the code, and read your post. Say I want a Texture resource manager, I would derive it from the resource manager class. The resource magaer class would contain load, delete, ect virtual functions, to be dealt with if needed in the derived classes. These other managers would contain thier own code to deal with meshes, textures, sounds ect. They would then load them into thier own vectors or arrays (I have seen std::map in ogre's manager but have never seen it before - what does it do?).
Then would you need a resource manager - manager? If I only wanted one class where I could go Mngr->Load(TEXTURE, "filename");(TEXTURE is enumed) Then would this class contain instances or pointers to the other managers?
Still have the question for finding the resource in the managers, should the name be just the filename and path? I see no other way for class 1 creating a texture and then class 4 needing/wanting to load the same texture (without knowing about class 1)
in yours, Mgr->GetGraphic( "graphic_name", "my_name"); what are the names signifying?
Hopefully I am starting to get it [wink]
Yea, after I started looking at Ogre's resource manager, I figured it must have to be abstract. But this brings up new questions.
I think I am starting to usderstand the more I looked at the code, and read your post. Say I want a Texture resource manager, I would derive it from the resource manager class. The resource magaer class would contain load, delete, ect virtual functions, to be dealt with if needed in the derived classes. These other managers would contain thier own code to deal with meshes, textures, sounds ect. They would then load them into thier own vectors or arrays (I have seen std::map in ogre's manager but have never seen it before - what does it do?).
Then would you need a resource manager - manager? If I only wanted one class where I could go Mngr->Load(TEXTURE, "filename");(TEXTURE is enumed) Then would this class contain instances or pointers to the other managers?
Still have the question for finding the resource in the managers, should the name be just the filename and path? I see no other way for class 1 creating a texture and then class 4 needing/wanting to load the same texture (without knowing about class 1)
in yours, Mgr->GetGraphic( "graphic_name", "my_name"); what are the names signifying?
Hopefully I am starting to get it [wink]
I have a separate module for D3D things and logically it has it's own resource management which is exposed to the engine. Although, I never found too much neede to unify that particular interface.
On the engine side, I have a tree structure which contains description of practically everything used in the program, for example, the about models and their animations but also about the program structure, scene entities such as vehicles, characters.
With the information stored in the tree I am able to create that particular object, when it is needed - and release when not needed - and recreate ... etc. Of course this tree doesn't contain information about the textures since they are managed by the graphics API.
Just going to throw my own experience with my engine here.
The way I did a resource manager, I abstracted the concept in this phrase: "a module respnsible for loading data from the drive and processing it, so that it fits the engine's internal data structure for that data type.".
Data types can be shaders scripts, lua scripts, mesh data, sound data, texture data, etc...
How does the engine know which Resource Loader to call? Simple, by looking at the file extension of the resource you're trying to load.
The engine has an internal Registry, and you can register a loader simply:
- A pointer to a file class, so that the function can read the file
- A pointer to a data structure, that the function is responsible for correctly filling up
Using this method, my plugin system allows plugins to register their own resource importers/exporters and data filters.
To export a resource that is already in mmory in the engine, all I need to do is call a registered "Res_Save" function, and pass to it the file class pointer and a data structure pointer.
Using this simple scheme, I've loaded ASE files into the engine, calculated the mesh's normals, and other atributes, TriStripify the mesh data, and then export using my own XM0 file format.
A resource loading function returns true or false if the data was successfully loaded onto the engine. In case it returns true, another module is responsible for adding the file and its now-in-memory data to a list of "already loaded resources".
Before calling a Resource Loader funtion, it checks if that resource has already been loaded, very simple design.
Hope it helped. [wink]
The way I did a resource manager, I abstracted the concept in this phrase: "a module respnsible for loading data from the drive and processing it, so that it fits the engine's internal data structure for that data type.".
Data types can be shaders scripts, lua scripts, mesh data, sound data, texture data, etc...
Quote:File --> Resource Loader --> Data Filter(s) --> Memory StructureThe data is read by the apropriate Resource Loader for that file format, then it might be post-load processed by a Data Filter (i have a DF for Meshes, that calculates normal data if it isn't present).
How does the engine know which Resource Loader to call? Simple, by looking at the file extension of the resource you're trying to load.
The engine has an internal Registry, and you can register a loader simply:
Quote: Register( type, extension, function_ptr );The above code registers a certain function as a resource loader for a certain file extension. When the function gets called, it receives two parameters:
Register( "Res_Load", "3ds", res_load_3ds );
- A pointer to a file class, so that the function can read the file
- A pointer to a data structure, that the function is responsible for correctly filling up
Using this method, my plugin system allows plugins to register their own resource importers/exporters and data filters.
To export a resource that is already in mmory in the engine, all I need to do is call a registered "Res_Save" function, and pass to it the file class pointer and a data structure pointer.
Using this simple scheme, I've loaded ASE files into the engine, calculated the mesh's normals, and other atributes, TriStripify the mesh data, and then export using my own XM0 file format.
A resource loading function returns true or false if the data was successfully loaded onto the engine. In case it returns true, another module is responsible for adding the file and its now-in-memory data to a list of "already loaded resources".
Before calling a Resource Loader funtion, it checks if that resource has already been loaded, very simple design.
Hope it helped. [wink]
Hmm, I didn't mention about the resource loading thing.
I explained a bit about my descriptor system but the benefit of the system is that it doesn't require any extra loader system (although I have that kind of "register-loader" system for loading certain things, but they aren't related).
The idea is that when a resource data is needed, it is asked from the corresponding descriptor. If the (resource) data is already loaded then a pointer to it is returned, otherwise the class described by the descriptor is created and initialised by the descriptor.
So for example, when a data of a model descriptor is requested and the data hasn't been loaded yet, the program creates a new model object (through a class factory) and then passes a pointer of the descriptor to the "Create" function. The model object has now a chance to load any data it might need. So there is no need for a separate loader in this case, the class knows itself how to set things up.
So might argue about the unsafety of the pointer conversion, but a similar method is used also in the famous 3d modelling software.
I explained a bit about my descriptor system but the benefit of the system is that it doesn't require any extra loader system (although I have that kind of "register-loader" system for loading certain things, but they aren't related).
The idea is that when a resource data is needed, it is asked from the corresponding descriptor. If the (resource) data is already loaded then a pointer to it is returned, otherwise the class described by the descriptor is created and initialised by the descriptor.
So for example, when a data of a model descriptor is requested and the data hasn't been loaded yet, the program creates a new model object (through a class factory) and then passes a pointer of the descriptor to the "Create" function. The model object has now a chance to load any data it might need. So there is no need for a separate loader in this case, the class knows itself how to set things up.
So might argue about the unsafety of the pointer conversion, but a similar method is used also in the famous 3d modelling software.
Author
I think I am starting to understand. I have translated what I think has been said/and tried to pseudo code it.
So now, to use the data:
Am I getting on track??
Still have questions about how to hold the data.
1) will I need template or something to have the actual data stay in the derived class? how can I have generic containers to put lets say IDirect3DTexture9 in one and ID3DXMesh in another?
2)what should organize the data? std::map or std::hash_map? or both. I an guessing the name of the resource will be it's filename, thus it would be an easy search
3) creating a new type of resource manager, would there be a better way (such as passing in only the name) of creating a new instance of a resource manager over passing the refrence in?
Many thanks on the replies
[Edited by - Valor Knight on March 30, 2006 5:33:23 PM]
class IResourceManager { private: protected: /* Resource array or something - what should it be (vector, list, map or what)? will it have to be a template to allow generic-ness? */ public: IResourceManager(void); virtual ~IResourceManager(void); virtual void create(); // this will be used by all other derived classes to load and take care of it thier own way //will add the resource to the vector, array, map or whatever virtual void unload( ulong handle ); virtual void unload( const std::wstring &name ); virtual void unloadall(); virtual void reloadall(); //Could be useful with lost devices virtual void remove( ulong handle ); virtual void remove( const std::wstring &name ); virtual void removeall(); }; class TextureResource : public IResourceManager { private: public: bool create(); //passes inputs to loadTexture, adds to list if created succesfully bool loadtexture(/*params to load a texture file, load the texture and if successful, returns true and create will add it to the resource list*/); // other functions as needed }; class Resources { private: std::vector<IResourceManager *> _ResourceManagers; //I guess this would have to be a map or hash map to add in the Id ("Texture") to be searched/refrenced easily. public: bool Register(/* ... */); bool Load();//calls create() for appropriate class bool Request(); };So now, to use the data:
say Mngr is an instance of Resources. Mngr->Register("Texture", TextureResource(passing in new()-ed) ); // adds a TextureResource class into the resource types - ID = Texture Mngr->AddFileType("Texture", ".dds"); // going on the topic of being allowed to either declare it is a texture object, or it will look for the extension in some game class: *texture1 = Mngr->Load("Texture", "filename.dds"); //loads into the "Texture" resource manager *texture2 = Mngr->Load("filename2.dds");//it is regestered so .dds is sent to the "Texture" manager //If it already exists, return the pointer to the data, otherwise, it will create it and return the pointerAm I getting on track??
Still have questions about how to hold the data.
1) will I need template or something to have the actual data stay in the derived class? how can I have generic containers to put lets say IDirect3DTexture9 in one and ID3DXMesh in another?
2)what should organize the data? std::map or std::hash_map? or both. I an guessing the name of the resource will be it's filename, thus it would be an easy search
3) creating a new type of resource manager, would there be a better way (such as passing in only the name) of creating a new instance of a resource manager over passing the refrence in?
Many thanks on the replies
[Edited by - Valor Knight on March 30, 2006 5:33:23 PM]
Here are a few more things to consider. Again, i'm not recommending one over the other. Just more approaches you can consider.
1) use a universal interface. You could go with operations like this:
Resource* GetResource(RESOURCE_TYPE, "name");
The gotcha here is that you get a generic Resource* in return. You would have to cast that. That's dumb. It's a nice interface though!
Workaround:
Templates. Yes, that's where it's really at! Templates are almost totally overlooked by even experienced programmers it seems. I'm just warming up to them myself. Here, for instance, you could have a template member function that looks like this:
TYPE* GetResource<TYPE>("name");
Then, though some kind of magic, you could select the right resource from the right list of resources.
2) Monolithic approach. Have one resource manager class and have a lot of functions:
Graphic* GetGraphic("name");
Sound* GetSound("name");
Font* GetFont("name");
Mesh* GetMesh("name");
PRO: it's all in one place. Oh, so simple.
CON: you have to recompile the manager everytime you futs with the resource types. BUT, usually you know what types you need at the beginning of the project. They are pretty obvious. It's not really the kind of thing you update all the time.
This is actually the approach i've taken. It's not as flexible, but like i said, you shouldn't have to flex it very often.
3) Separate managers. You already touched on this. Personally, i'm not too keen on it, although it does make things nicely modular. I like one-stop-shopping resource managers though ;-)
-----------------
Other things to try:
- you asked how you actually would store the resources internally. Most usually use a map, like so:
map< string, Resource* >;
I personally have a different map for each resource type. SoundMap, TextureMap, FontMap, etc...
- XML! I actually name and configure all my resources externally using XML (through TinyXML lib). With this, you can tell your system ahead of time what you want loaded (or at least make loadable), what the file names are, and what the aliases are. For instance, i can have an entry like this:
<TEXTURE name="Smoke Puff" file="foo/xyz.png">
<ALIAS name="enemy dies" />
<ALIAS name="gun fires" />
<ALIAS name="brick falls" />
</TEXTURE>
Now the resource manager loads this and links the 3 aliases with this filename. When any of the aliases are requested, it loads the file and returns the texture. Note here that the same texture is used for 3 different things! You can do this with sound effects and fonts and all the other resources as well.
- Hide implimentation details. You are best advised to create your own classes for resources. For instance, if you want sound fx, create your own SoundEffect class instead of just using your library's native structure (like SDL_MixChunk* or whatever). Hide all the library specific business underneath your resource and management classes.
The reason this is important is because you can completely withstand any changes you make if you change libraries. For instance, if you switched from SDL to FMOD for sound, you would have to change your loading and dumping code only. All the rest of your code that actually uses your homebrewed SoundEffect class would be totally untouched by such a huge change!
1) use a universal interface. You could go with operations like this:
Resource* GetResource(RESOURCE_TYPE, "name");
The gotcha here is that you get a generic Resource* in return. You would have to cast that. That's dumb. It's a nice interface though!
Workaround:
Templates. Yes, that's where it's really at! Templates are almost totally overlooked by even experienced programmers it seems. I'm just warming up to them myself. Here, for instance, you could have a template member function that looks like this:
TYPE* GetResource<TYPE>("name");
Then, though some kind of magic, you could select the right resource from the right list of resources.
2) Monolithic approach. Have one resource manager class and have a lot of functions:
Graphic* GetGraphic("name");
Sound* GetSound("name");
Font* GetFont("name");
Mesh* GetMesh("name");
PRO: it's all in one place. Oh, so simple.
CON: you have to recompile the manager everytime you futs with the resource types. BUT, usually you know what types you need at the beginning of the project. They are pretty obvious. It's not really the kind of thing you update all the time.
This is actually the approach i've taken. It's not as flexible, but like i said, you shouldn't have to flex it very often.
3) Separate managers. You already touched on this. Personally, i'm not too keen on it, although it does make things nicely modular. I like one-stop-shopping resource managers though ;-)
-----------------
Other things to try:
- you asked how you actually would store the resources internally. Most usually use a map, like so:
map< string, Resource* >;
I personally have a different map for each resource type. SoundMap, TextureMap, FontMap, etc...
- XML! I actually name and configure all my resources externally using XML (through TinyXML lib). With this, you can tell your system ahead of time what you want loaded (or at least make loadable), what the file names are, and what the aliases are. For instance, i can have an entry like this:
<TEXTURE name="Smoke Puff" file="foo/xyz.png">
<ALIAS name="enemy dies" />
<ALIAS name="gun fires" />
<ALIAS name="brick falls" />
</TEXTURE>
Now the resource manager loads this and links the 3 aliases with this filename. When any of the aliases are requested, it loads the file and returns the texture. Note here that the same texture is used for 3 different things! You can do this with sound effects and fonts and all the other resources as well.
- Hide implimentation details. You are best advised to create your own classes for resources. For instance, if you want sound fx, create your own SoundEffect class instead of just using your library's native structure (like SDL_MixChunk* or whatever). Hide all the library specific business underneath your resource and management classes.
The reason this is important is because you can completely withstand any changes you make if you change libraries. For instance, if you switched from SDL to FMOD for sound, you would have to change your loading and dumping code only. All the rest of your code that actually uses your homebrewed SoundEffect class would be totally untouched by such a huge change!
As a disclaimer, I've skipped everything everyone else has written in this thread.
When I was doing this, there were a couple goals I had in mind:
I think that covers the main points of my design.
When I was doing this, there were a couple goals I had in mind:
- Black box. The people using a resource are not necessarily entitled to know what that resource's interface is. The only knowledge they've been granted is that it exists under a certain class name.
- Reference counts are necessary, so that the system can know what objects are in use.
- Time stamps are necessary for streaming purposes. What this means in a practical sense is that the system needs to trap every time a resource is used.
- While string identifiers are useful, string lookups should be made rare.
- Access needed to be indirect, again due to streaming concerns.
- All of the resource types are known at compile time, so there is no need for dynamic/runtime type voodoo.
I think that covers the main points of my design.
Just a slight caveat before I begin, Resource Managers are one of those things that you'll design differently as you gain experience. Maybe don't mind so much if yours isn't totally swanky to start out with. It'll get better. The important thing is that it works, so you can get on with coding the game [and learning how your resource manager rocks/sucks]...
Ahem. That said, my current resource manager incarnation is a little different than most here. The key is resource definitions. A resource definition [in my setup] is simply a string/string pair. The key is the resource name, used like most here, and is pretty much the only thing available to the common user. The value is instructions about how to load the resource.
One of the major downsides to [most] of the methods listed above is it's near impossible to change loaders. If resources are moved into say a .pak file, or your sounds are now streamed... it's a big mess. With the resource definition method, only the value [the loading instructions] needs to be changed, and only in one config file.
I use a templated base resource manager to do that interpretation, and common tasks. Individual managers (sound,texture,font...) inherit from the base manager, and specify the constructed return type [returning raw resources is bad mmmkay
(Why?
Because you can't inherit from raw resources. If you return say... an image rather than a texture, you can later return varieties of things that inherit from an image.
Because you generally can't do proper reference counting semantics with a raw resource.
Because you shouldn't trust the programmer to do [or remember to do] the right things with the resource you return.
)].
The common base allows for common elements to be shared, but still allow for more specific specialization for resources that don't quite fit. It also allows for API specific implimentations to be done seperate from the rest, or even from their base manager.
Ahem. That said, my current resource manager incarnation is a little different than most here. The key is resource definitions. A resource definition [in my setup] is simply a string/string pair. The key is the resource name, used like most here, and is pretty much the only thing available to the common user. The value is instructions about how to load the resource.
One of the major downsides to [most] of the methods listed above is it's near impossible to change loaders. If resources are moved into say a .pak file, or your sounds are now streamed... it's a big mess. With the resource definition method, only the value [the loading instructions] needs to be changed, and only in one config file.
I use a templated base resource manager to do that interpretation, and common tasks. Individual managers (sound,texture,font...) inherit from the base manager, and specify the constructed return type [returning raw resources is bad mmmkay
(Why?
Because you can't inherit from raw resources. If you return say... an image rather than a texture, you can later return varieties of things that inherit from an image.
Because you generally can't do proper reference counting semantics with a raw resource.
Because you shouldn't trust the programmer to do [or remember to do] the right things with the resource you return.
)].
The common base allows for common elements to be shared, but still allow for more specific specialization for resources that don't quite fit. It also allows for API specific implimentations to be done seperate from the rest, or even from their base manager.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
