A Simple Templated Resource Manager

This topic is 4353 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

Recommended Posts

Download rmanager.h Hi everyone, this is a simple C++ templated resource manager, I'm looking for feedback and in particular, need help making the template prototype simpler, currently it looks like this:
template<
class resource_type,
class cleaner_type,
cleaner_type cleaner>
class rmanager{...};


Share on other sites
Personally, I would replace all the parameters excluding the resource_type with a "Policies" template parameter. The Policies class can be a template class if you like, where you have typedefs to load_options, loader, &c.

E.g.:
struct DefaultLoadOptions {};struct DefaultLoader {  void load() {}};struct DefaultPolicies {  typedef DefaultLoadOptions load_options_type;  typedef DefaultLoader loader_type;  //  Etc..};// Here's a template version:template<  class LoadOptions = DefaultLoadOptions,  class Loader = DefaultLoader> class Policies {  typedef LoadOptions load_options_type;  typedef Loader loader_type;};template<  class Type,  class Policies = DefaultPolicies // or class Polices = Policies<>> class ResourceManager {  typedef typename Policies::loader_type loader_type; // Convenience typedef if you use loader_type a lot.  public:    void load() {      loader_type loader;      loader.load();    }};

Just my opinion, though...

jfl.

 Minor edits.

[Edited by - jflanglois on January 11, 2006 6:06:03 AM]

Share on other sites
A few random thoughts, in no particular order:

1. What is the use for int ids? It brings one additional level of indirection and IMO only complicates matter. You could be using pack* instead.

2. In default constructor, it should be
default_options_available = false

3. Using const std::string & as function arguments is pretty standard.

4. Consider allowing generator() to throw exceptions. A little reformat of load function and you're good to go.

5. How can you use generator and cleaner with functors? I can't see that with passing instance directly. If it's only for functions, why bother passing function type if it's always the same? You could restrict those parameters to functors only and then if you wanted to use a function, the user would have to hide it in some standard wrapper (from std).

Share on other sites
I see, maybe I'm implementing this ass-backwards, but in your example, you're instantiating a load object at load time, but if the loader is just a function pointer that won't work. Which is why, as I have made it, an instance of the loader is passed when the manager is created (the same goes for the cleaner).

What ends up happening is that the loader requires two arguments in the template because the first defines its type, and the second is the instance itself.

would it make more sense to do it this way;
class loader{    static res * load(string,opts);}class cleaner{    static void clean(res *);}// ...resource_manager<res, opts, loader, cleaner>{    res * load(string s,opts o){        return loader::load(s, o);    }    void clean(res * r){        cleaner::clean(r);    }// ...};

A final note: in my mind it doesn't make sense to roll the opts, loader, and cleaner into a policies class, because loader and options must be intrinsically tied to res, ie there cannot exist a default loader and options object. The only thing that can be defaulted is cleaner (which would be just to delete the pointer) but even so, I wouldn't create a policy object to replace a single template argument, I would just change the arguments so they looked more like this:
template<    class resource_type,    class options_type,    class loader,    class cleaner=DeleteIt>resource_manager{...};

Share on other sites
Quote:
 Original post by defferA few random thoughts, in no particular order:1. What is the use for int ids? It brings one additional level of indirection and IMO only complicates matter. You could be using pack* instead.

do you mean the named_resources and loaded_resources maps are std::map<std::string, pack*>?

EDIT: Done, updated.

Quote:
 2. In default constructor, it should bedefault_options_available = false

Fixed. Updated.

Quote:
 3. Using const std::string & as function arguments is pretty standard.

I never bought into that, because std::string provides copy-on-change, and passing by reference actually adds a level of indirection that is not there if you just pass the strings around themselves.

Quote:
 4. Consider allowing generator() to throw exceptions. A little reformat of load function and you're good to go.

EDIT: OOPS I see what you meant, fixed and updated ;)

Quote:
 5. How can you use generator and cleaner with functors? I can't see that with passing instance directly. If it's only for functions, why bother passing function type if it's always the same? You could restrict those parameters to functors only and then if you wanted to use a function, the user would have to hide it in some standard wrapper (from std).
Well, this is the weakest part of the code right now anyway so it's open to change.

The simple quesiton is using it with functors, if you pass an instance of a class that implements a functor that matches the requirements of the load function, then it will be used.

The problem that I'm crunching on is how do you allow the user to pass an instance of a class, not the type of a class, to be a generator for resources. As I said, this solution works, but it's horribly inelegant.

Am I missing something?

[Edited by - Symphonic on January 11, 2006 10:26:38 AM]

Share on other sites
Quote:
 Original post by SymphonicHi everyone, this is a simple C++ templated resource manager, I'm looking for feedback and in particular, need help making the template prototype simpler, currently it looks like this:template< class resource_type, class load_options_type, class loader_type, loader_type loader, class cleaner_type, cleaner_type cleaner>class rmanager{...};

I would suggest what you want is
template<    class resource_type,    class load_options_type,    class loader_type,    class cleaner_type>  class rmanager{  public:    rmanager(loader_type &loader, cleaner_type& cleaner);    // ...  private:    loader_type  m_loader;    cleaner_type m_cleaner;  };

Have compile-time stuff set at compile time, run-time stuff set at run time.

Share on other sites
Quote:
 Original post by SymphonicA final note: in my mind it doesn't make sense to roll the opts, loader, and cleaner into a policies class, because loader and options must be intrinsically tied to res, ie there cannot exist a default loader and options object.

How come you allow the user to pass an arbitrary options type, while admitting that it's got to be strongly tied to loader and/or resource classes? IMO it should be defined as internal typedef in loader class.

I would also put cleaner class functionality into loader class and drop the former completely. Those are so strongly tied together, that I cannot imagine changing one with no impact on another.

Quote:
 Original post by SymphonicAm I missing something?

What Bregma said ;)

Share on other sites
Quote:
Original post by Symphonic
Quote:
 Original post by deffer3. Using const std::string & as function arguments is pretty standard.

I never bought into that, because std::string provides copy-on-change, and passing by reference actually adds a level of indirection that is not there if you just pass the strings around themselves.

The technical term is copy-on-write, and the C++ standard allows it as an optimisation, it does not require it. In fact recently copy-on-write has been falling out of favour as an optimisation since in a multithreaded environment the additional locking required makes it a pessimisation instead. Out of my five compiler versions only two have a default std::string implementation using copy-on-write and one of those is an older version of a compiler where the newer version does not use copy-on-write by default.

Enigma

Share on other sites
Quote:
 Why do I need help;It does work right now, as-is, but I would like not to have to pass loader_type and cleaner_type as arguments to the template, ideally, those would be implicit in the types of loader and cleaner. So how can I eliminate those?

When I made a generic templated manager, I solved that problem though a function pointer - if the user did not specify/need a 'cleaner', I did not force that upon them and if they needed it, they then called SetProcessFunctionPtr(ProcessFuncPtr. Take a look at my code and you can see what I mean, I'd suggest you take a similar approach - I've found it to be the best so far in terms of flexibility and usability. Good luck with your manager!

Share on other sites
Quote:
 Original post by BregmaI would suggest what you want istemplate< class resource_type, class load_options_type, class loader_type, class cleaner_type> class rmanager{ public: rmanager(loader_type &loader, cleaner_type& cleaner); // ... private: loader_type m_loader; cleaner_type m_cleaner; };Have compile-time stuff set at compile time, run-time stuff set at run time.

Bregma, spot-on, I guess I was thinking with my spleen when I designed that.

Fixed & Updated. The new version behaves like this:
// res, opt, gen and clean are all class namesgen g; clean c;resource_manager<res, opt, gen&, clean&> rmgr(g,c);

Quote:
 Original post by defferHow come you allow the user to pass an arbitrary options type, while admitting that it's got to be strongly tied to loader and/or resource classes? IMO it should be defined as internal typedef in loader class.I would also put cleaner class functionality into loader class and drop the former completely. Those are so strongly tied together, that I cannot imagine changing one with no impact on another.

That is a sensible way forward, I'm thinking about something like a caretaker class that defines the resource and locators as well, and disappearing the options mechanism to make it part of the locators.

That way, say if the user wants to, the same texture could be loaded with different filtering under different names, if the locator class differentiates between them.

Perhaps omething that looks more like this:
class exampleCaretaker{public:    struct resource{...}    struct locator{} // must define operator < () for maps to work    resource * load(const locator & loc){...}    void free(resource * res){...}}template<class caretaker>class resource_manager{private:    //...    caretaker ct;    load(caretaker::locator loc){        return ct.load(loc);    }    release(resource * r){        ct.free(r);    }public:    //...    resource_manager(caretaker c):ct(c){}};

Share on other sites
Quote:
Original post by Enigma
Quote:
Original post by Symphonic
Quote:
 Original post by deffer3. Using const std::string & as function arguments is pretty standard.

I never bought into that, because std::string provides copy-on-change, and passing by reference actually adds a level of indirection that is not there if you just pass the strings around themselves.

The technical term is copy-on-write, and the C++ standard allows it as an optimisation, it does not require it. In fact recently copy-on-write has been falling out of favour as an optimisation since in a multithreaded environment the additional locking required makes it a pessimisation instead. Out of my five compiler versions only two have a default std::string implementation using copy-on-write and one of those is an older version of a compiler where the newer version does not use copy-on-write by default.

Enigma

I looked through my string implementation just now, another motivating point is that the copy-constructor for std::string is called whenever it is passed by value instead of by reference, which means an extra round of destruction as well when it's finished...

In retrospect that would eat up any amount of speed you gain by just copying the pointer embodied by the string class.

oh well :) time to update the code

[Edited by - Symphonic on January 11, 2006 2:50:00 PM]

Share on other sites
Awesome!

I implemented the caretaker idea, the result is fantastic, here's some quick sample code:
// An example caretakerclass intfactory{public:	typedef int resource;	typedef std::string locator;	int * load(std::string foo){		int * pi = new int;		if(!pi) return false;		pi = foo.length();		return pi;	}	void free(int * pi){		delete pi;	}};// usage:intfactory factory;resource_manager<intfactory> intmanager(factory);

Works like a charm, thanks a million guys, the online version is updated.

Share on other sites
New Update: managed_resource utility class

normally, if you have a resource in the resource_manager, looking up the resource through resource_manager.get(string) involves a string lookup in a map that is potentially quite large.

The managed_resource class lets you create what is essentially a pointer to the resource itself, allowing very fast access as long as you keep an instance of the managed_resource around.

Example:
#include <rmanager.h>using namespace spforge;// ...// assume we have already defined our caretaker class, 'factory'// it loads things out of files, so the locators are just filenames (std::string)fix(factory::resource thing); // a function that we will feed things intofactory f;resource_manager<factory> rmgr(f);rmgr.load("foo", "data/foo.thing");managed_resource<factory> bar(rmgr, "data/bar.thing"); // loads the resource 'data/bar.thing'managed_resource<factory> foo("foo", rmgr); // uses the resource under the name 'foo' in rmgrfix(rmgr.get("foo")); // normal access method// you can use the managed resource as if it is a factory::resourcefix(bar); // new improved method!fix(foo); // sooo shiny!

The managed_resource objects integrate smoothly with the reference counting and automatic deallocation of resources.

Thoughts: Should I make it so that managed_resource doesn't have an independent public constructor, but is instead created by resource_manager?

Things I would like to add:
resource queries:
-> is this name registered?
-> how many references are there to this resource?
-> what are the available names for this resource?

-> if caretaker::locator supports the extraction (>>) operator, then any stream can be used to load a whole bunch of resources and assign multiple names to each one

For the automatic resource loading, if you have any ideas or insights about how to go about doing it, they would be appreciated.

Share on other sites
Quote:
 Original post by SymphonicNew Update: managed_resource utility classnormally, if you have a resource in the resource_manager, looking up the resource through resource_manager.get(string) involves a string lookup in a map that is potentially quite large.The managed_resource class lets you create what is essentially a pointer to the resource itself, allowing very fast access as long as you keep an instance of the managed_resource around.Example:[...]The managed_resource objects integrate smoothly with the reference counting and automatic deallocation of resources.

This sounds more like my own manager ;)
Ok, in the end you would want to use only the managed resource, and remove the pure resource* from your game code. You might consider using boost::shared_ptr or boost::intrusive_ptr for that, as they are already written for this purpose and are written well.

Quote:
 Original post by SymphonicThoughts: Should I make it so that managed_resource doesn't have an independent public constructor, but is instead created by resource_manager?

Sure:
managed_resource<factory> bar, foo;

and somewhere else:
foo = bar;
and at the end (releasing):
foo = 0;
bar = 0;

with all the ref-counting included all the time.

Quote:
 Original post by SymphonicThings I would like to add:resource queries:-> is this resource loaded?-> is this name registered?-> how many references are there to this resource?-> what are the available names for this resource?

YAGNI ?

Quote:
 Original post by SymphonicAutomatic resource loading using streams-> if caretaker::locator supports the extraction (>>) operator, then any stream can be used to load a whole bunch of resources and assign multiple names to each oneFor the automatic resource loading, if you have any ideas or insights about how to go about doing it, they would be appreciated.

Oh. Can you elaborate some more on this subject? How would you want to use this idea? Maybe a snippet of pseudocode?

Share on other sites
Quote:
 Original post by defferOk, in the end you would want to use only the managed resource, and remove the pure resource* from your game code. You might consider using boost::shared_ptr or boost::intrusive_ptr for that, as they are already written for this purpose and are written well.

If I'm reading you right, you mean that during every-frame usage of resources (like meshes for example) I will want to be using managed_resource objects to access the meshes, instead of using rmgr.get(string). This makes sense to me.

Quote:
 Sure:managed_resource bar, foo;bar = rmng.load("bar", "data/bar.jpg");foo = rmng.load("foo", "data/foo.png");and somewhere else:foo = bar;and at the end (releasing):foo = 0;bar = 0;with all the ref-counting included all the time.

That's only a little off, the thing is that managed_resource is associated with the resource_manager at instantiation (ie, there is no default constructor, only a constructor that takes a reference to the manager, and either an std::string as a name, or a caretaker::locator to directly load the resource).

Your code would look more like this:
rmng.load("bar", "data/bar.jpg");rmng.load("foo", "data/foo.png");managed_resource foo("foo", rmng), bar("bar", rmng);

Elsewhere:
foo = bar; // I did implement this, surprisingly

Release is handled by the destructors of the managed resources. So there's no need for explicitly releasing anything. This isn't a problem, unless you destroy the manager before you destroy the last managed_resource that points to it.

What I was thinking was something more along these lines:
rmng.load("bar", "data/bar.jpg");managed_resource res;res = rmng.getHandle("bar");

Here, the return type of rmng.getHandle is a pointer to a resource handle that is contained by rmng, which includes a reciprocal pointer back to res. That way, res can be used sensibly, and if rmng is destroyed before res, the destructor of rmng will go and neuter res so it can't do any damage (it will just throw an exception when access is attempted).

The question is this: Is that preferable, to how it is now, where the manager throws an exception from its destructor if there are still managed_resources out there?
Quote:
 YAGNI ?
probably right. Maybe just one: a function that returns the locator of the resource at name.
Quote:
 Oh. Can you elaborate some more on this subject? How would you want to use this idea? Maybe a snippet of pseudocode?

Ok, so this whole project was born out of a need for a texture manager and a mesh manager, I didn't want to write two, when I couldn't see how the core manager functionality could change between them.

So lets say I have a file that contains a list of all the meshes I want to load:
          Excerpt:data/my_mesh.3ds my_mesh name1 name2 ;data/your_mesh.3ds your_mesh ;data/our_mesh.3ds I don't share ;

So, it's pretty self explanatory, the first thing is the filename, and every word after them is a name for that resource, this can span multiple lines of course, because the list is terminated by a semicolon (;).

So here's the function I want to write for resource_manager:
void resource_manager::loadStream(std::istream & input){    caretaker::locator loc;    std::string name;    while(!input.eof()){        input >> loc; // requires that 'operator >> (istream&, caretaker::locator)' exists        input >> name;        while(name != ";"){            load(name, loc);            input >> name;        }    }}

The problem is, that the locators for the textures include a description of what type of filtering to use. So that would have to be included in the extraction operator for those objects, and then the texture resource file would look like this:
          Excerpt:data/textures/my_texture.tga TRI_LINEAR my_texture ;data/textures/font_texture.tga BI_LINEAR font ;

So I guess that that's ultimately in the hands of the person who implements the extraction operator for caretaker::locator, which is me anyway :D.

Share on other sites
Quote:
 Original post by SymphonicThat's only a little off, the thing is that managed_resource is associated with the resource_manager at instantiation (ie, there is no default constructor, only a constructor that takes a reference to the manager, and either an std::string as a name, or a caretaker::locator to directly load the resource).[...]Your code would look more like this:rmng.load("bar", "data/bar.jpg");rmng.load("foo", "data/foo.png");managed_resource foo("foo", rmng), bar("bar", rmng);

But that way you would have to ensure having valid device at instantiation (construction) time of every object containing any resource instance (also, if you implement your last idea, the config file must have been already handled). This could lead to some serious problems with order of initialization. I'm not saying that it would, but it could...

Quote:
 Original post by SymphonicHere, the return type of rmng.getHandle is a pointer to a resource handle that is contained by rmng, which includes a reciprocal pointer back to res. That way, res can be used sensibly, and if rmng is destroyed before res, the destructor of rmng will go and neuter res so it can't do any damage (it will just throw an exception when access is attempted).

If you're allowing managed_resource to be uninitialized, why bother disallowing it to have a default constructor in the first place? Also, I think you should simply disallow destroying the manager if any resource is unfreed (and report a leak accoordingly). In such case, the manager will be leaking as well, but who cares at that point.

IMHO, your idea of tracking all instances of managed_resource seems quite complex and is an another place for hard-to-track bugs.

Quote:
 Original post by SymphonicThe question is this: Is that preferable, to how it is now, where the manager throws an exception from its destructor if there are still managed_resources out there?

No, since destructors are not allowed to throw exceptions in the first place. You should report a leak and end gracefully. Also, I think some kind of release method should be used instead of simply deleting, where the manager would delete itself when no resources are left, but left otherwise (as I described above).

Quote:
 Original post by SymphonicConfig-file idea described

This idea sounds great. I guess I will implement it into my manager.
I only think that you don't need to use operator >> for that. A member function would be sufficient and make the code more clear to read.
Also, IIRC, using >> to read names restricts you to have single-word names.

EDIT:
You didn't try to compile you current code, did you? You should give it to some compiler to eat, as it lacks some typenames here and there. I don't know if anything else.

Share on other sites
Quote:
 Original post by defferBut that way you would have to ensure having valid device at instantiation (construction) time of every object containing any resource instance (also, if you implement your last idea, the config file must have been already handled). This could lead to some serious problems with order of initialization. I'm not saying that it would, but it could...

It could, and to be fair, I actually like it more that way, I feel like forcing the programmer to have a valid manager which exists before trying to create resources from it, and to free all externally referenced resources before the manager is destroyed is how it should be done.

Of course, that's a design decision more than an implementation problem, but important none-the-less; is it better to force a singleton resource manager, and hide it behind what is essentially a global interface, instead of having the programmer instantiate it when she wants and handling the resource_manager object itself.

In review I think that the singleton is a better approach, because it hides the resource management more sensibly, there would still have to be an initialization call to pass the caretaker to the manager, but beyond that it'd all be calls to static members and fiddling with managed_resources that automatically handle the singleton manager. That doesn't escape the requirement that the manager exists before any resource is created though.

Quote:
 No, since destructors are not allowed to throw exceptions in the first place. You should report a leak and end gracefully. Also, I think some kind of release method should be used instead of simply deleting, where the manager would delete itself when no resources are left, but left otherwise (as I described above).

How do I report a leak without throwing an exception? Do you mean using global error-tracking?

I like your idea about release(), if the manager is handled with a pointer instead of as a stack/global variable, that would work like a charm. So before the pointer is discarded, we called prmng->release() which tells the manager that once the last resource is freed to delete itself.

Quote:
 I only think that you don't need to use operator >> for that. A member function would be sufficient and make the code more clear to read. Also, IIRC, using >> to read names restricts you to have single-word names.
void resource_manager::loadStream(std::istream & in){    caretaker::locator loc;    std::string name;    char c;    while(in.good()){        loc = ct.readLocation(in); // reading a location is a property of caretaker        in.get(c);        while(iswhite(c)) in.get(c);        while(in.good() && c != ';'){            name.clear();            while(in.good() && c != ',' && c!= ';'){                name.push_back(c);                in.get(c);            }            if(name.length())load(name, loc);               }    }}

With this version, commas (,) are separators between names, and the sequence is ended with a semi-colon (;). If multi-line support or white-space contraction was needed it would be fairly trivial to add. And if you want to add names with leading whitespace you'd do it like this:
data/thing.jpg ,  foo,  bar;

Which is nifty.

Quote:
 You didn't try to compile you current code, did you? You should give it to some compiler to eat, as it lacks some typenames here and there. I don't know if anything else.

My compiler is VC++ 6.0 w/ STLPort, it compiles fine for me. If you get compile errors under your compiler please post them so I can have a look.
EDIT:I just double checked and I had not updated my online version to the most recent revision. Try again?

[Edited by - Symphonic on January 13, 2006 9:25:06 AM]

Share on other sites
Quote:
 Original post by SymphonicIt could, and to be fair, I actually like it more that way, I feel like forcing the programmer to have a valid manager which exists before trying to create resources from it, and to free all externally referenced resources before the manager is destroyed is how it should be done.[...more about singletons, which we do_not want to discuss here...]That doesn't escape the requirement that the manager exists before any resource is created though.

And how are you going to enforce this requiment?
For one thing, you practicaly won't be able to have any static objects using resources, because you won't be able to control the time and order of destruction for them.

Quote:
 Original post by SymphonicHow do I report a leak without throwing an exception? Do you mean using global error-tracking?

Ekhm... logging?

Quote:
 Original post by SymphonicMy compiler is VC++ 6.0 w/ STLPort, it compiles fine for me. If you get compile errors under your compiler please post them so I can have a look.EDIT:I just double checked and I had not updated my online version to the most recent revision. Try again?

I just PM'ed you the output from my compiler ;)

Share on other sites
Quote:
Original post by deffer
Quote:
 Original post by SymphonicThat doesn't escape the requirement that the manager exists before any resource is created though.

And how are you going to enforce this requiment?

Make it so that managed_resources cannot be constructed if (ie an exception is thrown if) the corresponding resource_manager has not already been initialized.

(I think I might have confused this in an earlier post, but the managed_resources don't have a default constructor, they must always point to a valid resource in a real manager)

I agree with you that worrying about chasing down deallocation is not something I want to do, so I will implement the release() idea, and have the resource_manager report any resources that are still allocated at the time of release() being called; which, it is expected (but not required), is the time at which a given manager is to be discarded).

Quote:
 Ekhm... logging?

Right, I thought you meant there was some way it could be dealt with at run time :) which, obviously, there isn't. Silly me.

Share on other sites

This topic is 4353 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

Create an account

Register a new account

• Forum Statistics

• Total Topics
628688
• Total Posts
2984243

• 16
• 13
• 13
• 10
• 10