A Simple Templated Resource Manager

Started by
17 comments, last by Symphonic 18 years, 3 months ago
Quote:Original post by Enigma
Quote:Original post by Symphonic
Quote:Original post by deffer
3. Using const std::string & as function arguments is pretty standard.

I never bought into that, because std::string provides copy-on-change, and passing by reference actually adds a level of indirection that is not there if you just pass the strings around themselves.

The technical term is copy-on-write, and the C++ standard allows it as an optimisation, it does not require it. In fact recently copy-on-write has been falling out of favour as an optimisation since in a multithreaded environment the additional locking required makes it a pessimisation instead. Out of my five compiler versions only two have a default std::string implementation using copy-on-write and one of those is an older version of a compiler where the newer version does not use copy-on-write by default.

Enigma

I looked through my string implementation just now, another motivating point is that the copy-constructor for std::string is called whenever it is passed by value instead of by reference, which means an extra round of destruction as well when it's finished...

In retrospect that would eat up any amount of speed you gain by just copying the pointer embodied by the string class.

oh well :) time to update the code

[Edited by - Symphonic on January 11, 2006 2:50:00 PM]
Geordi
George D. Filiotis
Advertisement
Awesome!

I implemented the caretaker idea, the result is fantastic, here's some quick sample code:
// An example caretakerclass intfactory{public:	typedef int resource;	typedef std::string locator;	int * load(std::string foo){		int * pi = new int;		if(!pi) return false;		pi = foo.length();		return pi;	}	void free(int * pi){		delete pi;	}};// usage:intfactory factory;resource_manager<intfactory> intmanager(factory);

Works like a charm, thanks a million guys, the online version is updated.
Geordi
George D. Filiotis
New Update: managed_resource utility class

normally, if you have a resource in the resource_manager, looking up the resource through resource_manager.get(string) involves a string lookup in a map that is potentially quite large.

The managed_resource class lets you create what is essentially a pointer to the resource itself, allowing very fast access as long as you keep an instance of the managed_resource around.

Example:
#include <rmanager.h>using namespace spforge;// ...// assume we have already defined our caretaker class, 'factory'// it loads things out of files, so the locators are just filenames (std::string)fix(factory::resource thing); // a function that we will feed things intofactory f;resource_manager<factory> rmgr(f);rmgr.load("foo", "data/foo.thing");managed_resource<factory> bar(rmgr, "data/bar.thing"); // loads the resource 'data/bar.thing'managed_resource<factory> foo("foo", rmgr); // uses the resource under the name 'foo' in rmgrfix(rmgr.get("foo")); // normal access method// you can use the managed resource as if it is a factory::resourcefix(bar); // new improved method!fix(foo); // sooo shiny!

The managed_resource objects integrate smoothly with the reference counting and automatic deallocation of resources.

Thoughts: Should I make it so that managed_resource doesn't have an independent public constructor, but is instead created by resource_manager?

Things I would like to add:
resource queries:
-> is this resource loaded?
-> is this name registered?
-> how many references are there to this resource?
-> what are the available names for this resource?

Automatic resource loading using streams
-> if caretaker::locator supports the extraction (>>) operator, then any stream can be used to load a whole bunch of resources and assign multiple names to each one

For the automatic resource loading, if you have any ideas or insights about how to go about doing it, they would be appreciated.
Geordi
George D. Filiotis
Quote:Original post by Symphonic
New Update: managed_resource utility class

normally, if you have a resource in the resource_manager, looking up the resource through resource_manager.get(string) involves a string lookup in a map that is potentially quite large.

The managed_resource class lets you create what is essentially a pointer to the resource itself, allowing very fast access as long as you keep an instance of the managed_resource around.

Example:
[...]
The managed_resource objects integrate smoothly with the reference counting and automatic deallocation of resources.


This sounds more like my own manager ;)
Ok, in the end you would want to use only the managed resource, and remove the pure resource* from your game code. You might consider using boost::shared_ptr or boost::intrusive_ptr for that, as they are already written for this purpose and are written well.

Quote:Original post by Symphonic
Thoughts: Should I make it so that managed_resource doesn't have an independent public constructor, but is instead created by resource_manager?


Sure:
managed_resource<factory> bar, foo;
bar = rmng.load("bar", "data/bar.jpg");
foo = rmng.load("foo", "data/foo.png");
and somewhere else:
foo = bar;
and at the end (releasing):
foo = 0;
bar = 0;
with all the ref-counting included all the time.

Quote:Original post by Symphonic
Things I would like to add:
resource queries:
-> is this resource loaded?
-> is this name registered?
-> how many references are there to this resource?
-> what are the available names for this resource?

YAGNI ?

Quote:Original post by Symphonic
Automatic resource loading using streams
-> if caretaker::locator supports the extraction (>>) operator, then any stream can be used to load a whole bunch of resources and assign multiple names to each one

For the automatic resource loading, if you have any ideas or insights about how to go about doing it, they would be appreciated.


Oh. Can you elaborate some more on this subject? How would you want to use this idea? Maybe a snippet of pseudocode?
Quote:Original post by deffer
Ok, in the end you would want to use only the managed resource, and remove the pure resource* from your game code. You might consider using boost::shared_ptr or boost::intrusive_ptr for that, as they are already written for this purpose and are written well.

If I'm reading you right, you mean that during every-frame usage of resources (like meshes for example) I will want to be using managed_resource objects to access the meshes, instead of using rmgr.get(string). This makes sense to me.

(Reading boost stuff...)

Quote:Sure:
managed_resource<factory> bar, foo;
bar = rmng.load("bar", "data/bar.jpg");
foo = rmng.load("foo", "data/foo.png");
and somewhere else:
foo = bar;
and at the end (releasing):
foo = 0;
bar = 0;
with all the ref-counting included all the time.

That's only a little off, the thing is that managed_resource is associated with the resource_manager at instantiation (ie, there is no default constructor, only a constructor that takes a reference to the manager, and either an std::string as a name, or a caretaker::locator to directly load the resource).

Your code would look more like this:
rmng.load("bar", "data/bar.jpg");rmng.load("foo", "data/foo.png");managed_resource foo("foo", rmng), bar("bar", rmng);

Elsewhere:
foo = bar; // I did implement this, surprisingly

Release is handled by the destructors of the managed resources. So there's no need for explicitly releasing anything. This isn't a problem, unless you destroy the manager before you destroy the last managed_resource that points to it.

What I was thinking was something more along these lines:
rmng.load("bar", "data/bar.jpg");managed_resource res;res = rmng.getHandle("bar");

Here, the return type of rmng.getHandle is a pointer to a resource handle that is contained by rmng, which includes a reciprocal pointer back to res. That way, res can be used sensibly, and if rmng is destroyed before res, the destructor of rmng will go and neuter res so it can't do any damage (it will just throw an exception when access is attempted).

The question is this: Is that preferable, to how it is now, where the manager throws an exception from its destructor if there are still managed_resources out there?
Quote:YAGNI ?
probably right. Maybe just one: a function that returns the locator of the resource at name.
Quote:Oh. Can you elaborate some more on this subject? How would you want to use this idea? Maybe a snippet of pseudocode?


Ok, so this whole project was born out of a need for a texture manager and a mesh manager, I didn't want to write two, when I couldn't see how the core manager functionality could change between them.

So lets say I have a file that contains a list of all the meshes I want to load:
          Excerpt:data/my_mesh.3ds my_mesh name1 name2 ;data/your_mesh.3ds your_mesh ;data/our_mesh.3ds I don't share ;

So, it's pretty self explanatory, the first thing is the filename, and every word after them is a name for that resource, this can span multiple lines of course, because the list is terminated by a semicolon (;).

So here's the function I want to write for resource_manager:
void resource_manager::loadStream(std::istream & input){    caretaker::locator loc;    std::string name;    while(!input.eof()){        input >> loc; // requires that 'operator >> (istream&, caretaker::locator)' exists        input >> name;        while(name != ";"){            load(name, loc);            input >> name;        }    }}

The problem is, that the locators for the textures include a description of what type of filtering to use. So that would have to be included in the extraction operator for those objects, and then the texture resource file would look like this:
          Excerpt:data/textures/my_texture.tga TRI_LINEAR my_texture ;data/textures/font_texture.tga BI_LINEAR font ;

So I guess that that's ultimately in the hands of the person who implements the extraction operator for caretaker::locator, which is me anyway :D.
Geordi
George D. Filiotis
Quote:Original post by Symphonic
That's only a little off, the thing is that managed_resource is associated with the resource_manager at instantiation (ie, there is no default constructor, only a constructor that takes a reference to the manager, and either an std::string as a name, or a caretaker::locator to directly load the resource).
[...]
Your code would look more like this:
rmng.load("bar", "data/bar.jpg");rmng.load("foo", "data/foo.png");managed_resource foo("foo", rmng), bar("bar", rmng);



But that way you would have to ensure having valid device at instantiation (construction) time of every object containing any resource instance (also, if you implement your last idea, the config file must have been already handled). This could lead to some serious problems with order of initialization. I'm not saying that it would, but it could...

Quote:Original post by Symphonic
Here, the return type of rmng.getHandle is a pointer to a resource handle that is contained by rmng, which includes a reciprocal pointer back to res. That way, res can be used sensibly, and if rmng is destroyed before res, the destructor of rmng will go and neuter res so it can't do any damage (it will just throw an exception when access is attempted).


If you're allowing managed_resource to be uninitialized, why bother disallowing it to have a default constructor in the first place? Also, I think you should simply disallow destroying the manager if any resource is unfreed (and report a leak accoordingly). In such case, the manager will be leaking as well, but who cares at that point.

IMHO, your idea of tracking all instances of managed_resource seems quite complex and is an another place for hard-to-track bugs.


Quote:Original post by Symphonic
The question is this: Is that preferable, to how it is now, where the manager throws an exception from its destructor if there are still managed_resources out there?


No, since destructors are not allowed to throw exceptions in the first place. You should report a leak and end gracefully. Also, I think some kind of release method should be used instead of simply deleting, where the manager would delete itself when no resources are left, but left otherwise (as I described above).


Quote:Original post by Symphonic
Config-file idea described


This idea sounds great. I guess I will implement it into my manager.
I only think that you don't need to use operator >> for that. A member function would be sufficient and make the code more clear to read.
Also, IIRC, using >> to read names restricts you to have single-word names.

EDIT:
You didn't try to compile you current code, did you? You should give it to some compiler to eat, as it lacks some typenames here and there. I don't know if anything else.
Quote:Original post by deffer
But that way you would have to ensure having valid device at instantiation (construction) time of every object containing any resource instance (also, if you implement your last idea, the config file must have been already handled). This could lead to some serious problems with order of initialization. I'm not saying that it would, but it could...

It could, and to be fair, I actually like it more that way, I feel like forcing the programmer to have a valid manager which exists before trying to create resources from it, and to free all externally referenced resources before the manager is destroyed is how it should be done.

Of course, that's a design decision more than an implementation problem, but important none-the-less; is it better to force a singleton resource manager, and hide it behind what is essentially a global interface, instead of having the programmer instantiate it when she wants and handling the resource_manager object itself.

In review I think that the singleton is a better approach, because it hides the resource management more sensibly, there would still have to be an initialization call to pass the caretaker to the manager, but beyond that it'd all be calls to static members and fiddling with managed_resources that automatically handle the singleton manager. That doesn't escape the requirement that the manager exists before any resource is created though.

Quote:No, since destructors are not allowed to throw exceptions in the first place. You should report a leak and end gracefully. Also, I think some kind of release method should be used instead of simply deleting, where the manager would delete itself when no resources are left, but left otherwise (as I described above).


How do I report a leak without throwing an exception? Do you mean using global error-tracking?

I like your idea about release(), if the manager is handled with a pointer instead of as a stack/global variable, that would work like a charm. So before the pointer is discarded, we called prmng->release() which tells the manager that once the last resource is freed to delete itself.


Quote:I only think that you don't need to use operator >> for that. A member function would be sufficient and make the code more clear to read. Also, IIRC, using >> to read names restricts you to have single-word names.
How about something more scanner-like:
void resource_manager::loadStream(std::istream & in){    caretaker::locator loc;    std::string name;    char c;    while(in.good()){        loc = ct.readLocation(in); // reading a location is a property of caretaker        in.get(c);        while(iswhite(c)) in.get(c);        while(in.good() && c != ';'){            name.clear();            while(in.good() && c != ',' && c!= ';'){                name.push_back(c);                in.get(c);            }            if(name.length())load(name, loc);               }    }}

With this version, commas (,) are separators between names, and the sequence is ended with a semi-colon (;). If multi-line support or white-space contraction was needed it would be fairly trivial to add. And if you want to add names with leading whitespace you'd do it like this:
data/thing.jpg ,  foo,  bar;

Which is nifty.

Quote:You didn't try to compile you current code, did you? You should give it to some compiler to eat, as it lacks some typenames here and there. I don't know if anything else.

My compiler is VC++ 6.0 w/ STLPort, it compiles fine for me. If you get compile errors under your compiler please post them so I can have a look.
EDIT:I just double checked and I had not updated my online version to the most recent revision. Try again?

[Edited by - Symphonic on January 13, 2006 9:25:06 AM]
Geordi
George D. Filiotis
Quote:Original post by Symphonic
It could, and to be fair, I actually like it more that way, I feel like forcing the programmer to have a valid manager which exists before trying to create resources from it, and to free all externally referenced resources before the manager is destroyed is how it should be done.

[...more about singletons, which we do_not want to discuss here...]

That doesn't escape the requirement that the manager exists before any resource is created though.


And how are you going to enforce this requiment?
For one thing, you practicaly won't be able to have any static objects using resources, because you won't be able to control the time and order of destruction for them.


Quote:Original post by Symphonic
How do I report a leak without throwing an exception? Do you mean using global error-tracking?


Ekhm... logging?

Quote:Original post by Symphonic
My compiler is VC++ 6.0 w/ STLPort, it compiles fine for me. If you get compile errors under your compiler please post them so I can have a look.
EDIT:I just double checked and I had not updated my online version to the most recent revision. Try again?


I just PM'ed you the output from my compiler ;)
Quote:Original post by deffer
Quote:Original post by SymphonicThat doesn't escape the requirement that the manager exists before any resource is created though.


And how are you going to enforce this requiment?

Make it so that managed_resources cannot be constructed if (ie an exception is thrown if) the corresponding resource_manager has not already been initialized.

(I think I might have confused this in an earlier post, but the managed_resources don't have a default constructor, they must always point to a valid resource in a real manager)

I agree with you that worrying about chasing down deallocation is not something I want to do, so I will implement the release() idea, and have the resource_manager report any resources that are still allocated at the time of release() being called; which, it is expected (but not required), is the time at which a given manager is to be discarded).

Quote:Ekhm... logging?

Right, I thought you meant there was some way it could be dealt with at run time :) which, obviously, there isn't. Silly me.
Geordi
George D. Filiotis

This topic is closed to new replies.

Advertisement