A note on resources and managers, why it is so hard to write a good one?

Started by
48 comments, last by iblis 14 years, 9 months ago
This topic is killing me, I want it to be simple as possible but not simpler, I can't just think of a way to do it. Ill start with what I came to: This is Reference counted interface, objects that wishes to be reference counted must inherit from here

        /** 
	 * Interface of reference counted object.
	 */
	class SKGE_API IReferenceCounted{
	private:
		uint m_RefCounter; /**< Number of references. */

	protected:
		/** 
		 * Add Reference.
		 */
		inline void addRef() { m_RefCounter++; }

		/** 
		 * Remove reference.
		 * @return New reference value after decrement.
		 */
		inline uint removeRef() { m_RefCounter--; return m_RefCounter; }

	public:
		/** 
		 * Constructor.
		 */
		IReferenceCounted():m_RefCounter(1){}
	};

Now I have Resource that implement the reference counter:

	/** 
	* Resource Handle.
	*/
	typedef ulong ResourceHandle;

	/** 
	 * Abstract Resource Class.
	 */
	class SKGE_API Resource: public IReferenceCounted{

		friend ResourceManager;

	protected:
		String m_Name; /**< Resource name. */
		ResourceHandle m_Handle; /**< Resource Handle. */
		bool m_Created; /**< Indicates if the resource was created. */
		ResourceManager* m_Creator; /**< Manager that created current resource. */

	public:
		/** 
		 * Constructor. Sets resource information, but does not load it.
		 * @param _name Resource name.
		 * @param _handle Resource handle.
		 * @param _creator Manager that created current resource.
		 */
		Resource(const String& _name, ResourceHandle _handle, ResourceManager* _creator):
		  m_Name(_name),
		  m_Creator(_creator),
		  m_Handle(_handle),
		  m_Created(false)
	    {}

	    /** 
	     * Destructor. Does not destroy the resource.
	     */
		virtual ~Resource(){}

		/** 
		 * Was the resource created?
		 * @return True if resource was created, otherwise false.
		 */
		inline bool created() const { return m_Created; }

		/** 
		 * Get resource name.
		 * @return Resource name.
		 */
		inline const String& getName() const { return m_Name; }

		/** 
		 * Get resource handle.
		 * @return Resource Handle.
		 */
		inline ResourceHandle getHandle() const { return m_Handle; }
	};

Note: Resource manager is a friend of Resource to be able to do reference counting on it (if you noticed all IReferenceCounted's methods are protected), I don't like friend classes/function but making addRef()/removeRef() public IMHO brakes the encapsulation so I had to choose between two bad methods. And the last one is ResourceManager

        /** 
	 * Abstract resource manager.
	 */
	class SKGE_API ResourceManager{
	private:
		static ResourceHandle sNextResourceHandle; /**< Next available resource handle. */

	protected:
		/** 
		 * Resource maps.
		 */
		//@{
		typedef stdext::hash_map<String, Resource*> ResourceNameMap;
		typedef stdext::hash_map<ResourceHandle, Resource*> ResourceHandleMap;
		//@}

		ResourceNameMap m_NameResourceMap; /**< Name->Resource Map. */
		ResourceHandleMap m_HandleResourceMap; /**< Handle->Resource Map. */
		String m_ResourcesDirectory; /**< Directory where resources sits. */

		/** 
		 * Resource removal.
		 * Totally destroy and remove resource from the system.
		 * @param _resource Resource to remove and destroy.
		 */
		void _removeResource(Resource* _resource);

	public:
		/** 
		 * Constructor.
		 */
		ResourceManager(){}

		/** 
		 * Destructor. Delete all resources
		 */
		virtual ~ResourceManager();

		/** 
		 * Get resources directory.
		 * @return Directory where resources sits.
		 */
		inline const String& getResourcesDirectory() const { return m_ResourcesDirectory; }

		/** 
		 * Set Resources Directory.
		 * @param _dir Resources Directory.
		 */
		inline void setResourcesDirectory(const String& _dir) { m_ResourcesDirectory = _dir; }

		/** 
		 * Get resource by name or handle.
		 * @return Resource or 0 if does not exists.
		 */
		//@{
		Resource* acquire(const String& _name) const;
		Resource* acquire(ResourceHandle _handle) const;
		//@}

		/** 
		 * Release resource and if needed delete it from system.
		 * @param _resoruce Resource to release.
		 */
		void release(Resource* _resource);
	};

Most of it self-explanatory, just to note acquire will search in the appropriate hash map and if found it will increase the ref calling addRef() and will return in, release on the other side will decrement the ref by calling removeRef() and if it reaches 0, it will call _removeResource() to totally delete and remove the resource from both hash maps. Till this point everything is fine. But most of you will notice that there is some obligations that if wont be followed will cause memory leaking. Assume the following:

//Till this point resource was loaded
{
 Resource* res = ResourceManager->acquire("Res1");
 //Do some stuff with resource

 //And here we must call this
 ResourceManager->release(res);
}

Ugly ain't it? Too complicated, as user of this system I don't care about the way of freeing resources, especially via resource manager (there will be more that 1, for textures, meshes, fonts, particles, etc). I could simply this by adding the following to Resource class:

void release(){
 m_Creator->release(this);
}

(Assuming the resource was loaded by the manager, in other cases I would say RTFM :] ) Now the following code could be replaced to this one:

//Till this point resource was loaded
{
 Resource* res = ResourceManager->acquire("Res1");
 //Do some stuff with resource

 //And here we must call this
 res->release();
}

Nicer right? But still not good, I still have to remember to free the resource. So I thought about Smart Pointers, something like:

//In resource manager change acquire to this:
 MageSharedPtr<Resource*> acquire(...);

//and the write like this
//Till this point resource was loaded
{
 MegaSharedPtr<Resource*> res = ResourceManager->acquire("Res1");
 //Do some stuff with resource

 //And we don't need to care about freeing it, cause the destructor
 //of MegaSharedPtr will call release() of the Resource*
}

And this is nice! But it brakes when I have to expand my resource to a more generalized resources like Texture/Mesh. Ill have to write conversions from MegaSharedPtr<Resource*> to MegaSharedPtr<Texture*> to be able to do it normally, cause Load() of Texture class would like to return Texture*, and acquire returns MegaSharedPtr<Resource*> that Ill have to cast to MegaSharedPtr<Texture*> or MegaSharedPtr<Mesh*> or etc.. So I'm kinda confused here and don't know how to do it right, it looks too complicated and I don't know how to simplify that :( I would like to have some suggestions or articles about the topic. Thanks a lot!

I would love to change the world, but they won’t give me the source code.

Advertisement
Quote:Original post by s.kwee
But it brakes when I have to expand my resource to a more generalized resources like Texture/Mesh.
Ill have to write conversions from MegaSharedPtr<Resource*> to MegaSharedPtr<Texture*> to be able to do it normally, cause Load() of Texture class would like to return Texture*, and acquire returns MegaSharedPtr<Resource*> that Ill have to cast to MegaSharedPtr<Texture*> or MegaSharedPtr<Mesh*> or etc..

So I'm kinda confused here and don't know how to do it right, it looks too complicated and I don't know how to simplify that :(
I would like to have some suggestions or articles about the topic.

Thanks a lot!


If you template your resource manager then your "get" function can do the type cast for you. Then you would only have to do something like Texture *texture = Manager<Texture>.Get("resource") or Mesh *mesh = Manager<Mesh>.Get("resource").

Yes its nice, but since every resource is loaded differently (texture would get as parameter the file name, a boolean that says if there is a need to generate mip-maps, and probably a boolean to say if there is a need to clamp or repeat the texture, font on the other side will get file name and font size as parameters, particle will get the filename and time to live (for example)) so they all different and there is nothing common (except the get and release methods).

I would love to change the world, but they won’t give me the source code.

1. Do not use your own reference counting. boost::shared_ptr is good. boost::weak_ptr is very useful with resource managers. boost::intrusive_ptr if you *really* want that. They're damned near standard too.

2. Assuming your resources will exist in a file is naive. Assuming they're all going to be in the same directory is just kind of criminally limiting your design. Not the biggest thing now, but...

3. Templated managers as stupid_programmer suggests are good, and will solve the root problem that you're asking about. It also allows you to inherit from the base manager and do specialization for sounds/textures/etc.
Quote:1. Do not use your own reference counting. boost::shared_ptr is good. boost::weak_ptr is very useful with resource managers. boost::intrusive_ptr if you *really* want that. They're damned near standard too.

I'm pretty anti-boost, I have my own conclusions about this library, and for now prefer not to use it.

Quote:2. Assuming your resources will exist in a file is naive. Assuming they're all going to be in the same directory is just kind of criminally limiting your design. Not the biggest thing now, but...

As I move, new things come to my mind and I add them later, I can't think about everything now, since I also learn during the process. And I didn't say all resource is located in a file ;) For example in Texture Manager Ill of course provide two methods: createFromFril, createRaw. Also if you noticed about the variable m_ResourcesDirectory and methods to get/set it, so you kinda can change the folder :)

Quote:3. Templated managers as stupid_programmer suggests are good, and will solve the root problem that you're asking about. It also allows you to inherit from the base manager and do specialization for sounds/textures/etc.

You mean something like:
template<class T>class ResourceManager{  T* get();  void releat(T*);};class TextureManager: public ResourceManager<Texture>{ Texture* load(); ....};class MeshManager: public ResourceManager<Mesh>{ Mesh* load(); ....};

Yes? I thought about this way, but C++ lacks of the inheritable templates like Java have, there for I can't limit the template to be a sub class of resource and could write something like:
class AbcManager: public ResourceManager<Abc>{};

Where Abc is not a sub class of Resource, and by this logically I make resource manager from being a resource manager to anything manager. It's not bad I just dislike the style where you can't see lines between what you have and what you can have. However I will consider this as option because I'm really out of ideas.

Thanks again!

I would love to change the world, but they won’t give me the source code.

As far as casting, this is what i'm rollin with..

/// The ResourceHandle wraps a shared_ptr to a Resource. The Resource is stored as a pointer///  to the base class. The accessor function operator-&gt;() and Get() will cast the Resource///  to the templated type of the ResourceHandle. Checks are made to ensure that the ResourceHandle///  will only store a Resource of the templated type.///template &lt;typename T&gt;class ResourceHandle{public:	ResourceHandle()	{ }	ResourceHandle(boost::shared_ptr&lt;Resource&gt; &resource)	{		if (!resource.get()) return;		Assert(resource-&gt;GetTypeID() == T::GetTypeIDStatic(), "ResourceHandle type mismatch.");		m_P = resource;	}	ResourceHandle(const ResourceHandle &rh)	{		operator = (rh.m_P);	}	void operator = (const ResourceHandle &rh)	{		operator = (rh.m_P);				}	void operator = (const boost::shared_ptr&lt;Resource&gt; &resource)	{		if (!resource.get()) return;		Assert(resource-&gt;GetTypeID() == T::GetTypeIDStatic(), "ResourceHandle type mismatch.");		m_P = resource;	}	T *      operator-&gt;()       { return static_cast&lt;T*&gt;(m_P.get()); }	const T *operator-&gt;() const { return static_cast&lt;T*&gt;(m_P.get()); }	T *      Get()       { return static_cast&lt;T*&gt;(m_P.get()); }	const T *Get() const { return static_cast&lt;T*&gt;(m_P.get()); }	bool   IsSet() const { return m_P.get() != NULL; }	bool isStatus(ResourceStatus status) const { return m_P.get() ? m_P-&gt;Status() == status : false; }private:	boost::shared_ptr&lt;Resource&gt; m_P;};


As far as the different data required to construct a Resource. The Resource object has a Load function which takes a File pointer. This file could be a binary file, or an xml node. If it's not what the concrete resource expected then it complains, otherwise it just extracts the necessary data.

In other words, the depending information is injected into the Resource via an abstract interface File.
bzroom
Thats nice, solves some problems! But do you have any kind of Resource Manager? How do you ensure that you have only single instance of particular resource in memory?

I would love to change the world, but they won’t give me the source code.

Quote:Original post by s.kwee
Quote:1. Do not use your own reference counting. boost::shared_ptr is good. boost::weak_ptr is very useful with resource managers. boost::intrusive_ptr if you *really* want that. They're damned near standard too.

I'm pretty anti-boost, I have my own conclusions about this library, and for now prefer not to use it.


I submit that your own conclusions are wrong, at least regarding the shared pointer portion of the library. I've heard one mildly viable argument against that portion of the library, and I'm fairly certain it doesn't apply in your case. So what's the reasoning?


Quote:
Quote:3. Templated managers as stupid_programmer suggests are good, and will solve the root problem that you're asking about. It also allows you to inherit from the base manager and do specialization for sounds/textures/etc.

You mean something like:
*** Source Snippet Removed ***
Yes? I thought about this way, but C++ lacks of the inheritable templates like Java have, there for I can't limit the template to be a sub class of resource and could write something like:
*** Source Snippet Removed ***
Where Abc is not a sub class of Resource


So what? What common behavior do all resources require? Will you ever need to store homogeneous collections of Resources together?

Quote:
It's not bad I just dislike the style where you can't see lines between what you have and what you can have.


There's no reason not to allow the things you're mentally trying to exclude.
Every time I see a question like this I wish the article I'm writing for Gamedev on my resource manager were complete [grin]

Without going into too many specifics, I can point out a couple ares where you could improve your manager.

Firstly, "friend" is a huge red flag -- as you are aware -- that the design is flawed. In fact, "friend" is the absolute strongest relationship that can be expressed in C++, stronger even than inheritance, and as such, its power should only be called upon when absolutely necessary and justified. To require that any class which manipulates a refcount to be a friend of the ref-counted class is like using a sledgehammer to hang a picture-nail.

The ultimate issue you have is that your model of resources violates the single responsibility principle -- it is this poor separation of responsibilities that is leading to your problems. Your resource seems to "describe" the resource (in your case, naming it), hold its handle, hold its creation data (manager and created), and to presumably also act as the run-time data source for the resource (your example doesn't show this.)

In a well-separated system, such as mine (if I may be so bold), the separation of responsibilities breaks down something like this:

ResourceInfo - Identifies a resource within the system. This may be a filename, unique string, GUID, or other id.

ResourceData - Provides the data that the resource holds. The data (such as a texture or sound resource) doesn't and shouldn't know anything about how, or even whether it is "managed" -- it should be a class capable of standing on its own.

A Handle - The handle is what a client holds, and which allows access to the ResourceData instance associated with the handle. It is the handle which owns the ResourceData instance, and it is the handle which is ref-counted (multiple handles exist, not multiple copies of the resource). In my system, shared_ptr serves as the handle. shared_ptr is part of the tr1 extensions to the C++ standard library, and can be provided by boost if your platform does not yet support tr1, so this is a standards-friendly way of providing shared references to an object.

A Manager - The manager is what contains a distinct collection of resources. It provides methods for acquiring a resource handle (initiating loading if necessary), testing resource availability, and freeing a resource. Part of its job entails providing a map from ResourceInfo to its associated handle -- in my system, where handle is ref-counted, this means that the manager owns the initial handle (and hence, owns the ResourceData instance indirectly). This means that disposal of a resource can only come through a method call on the manager -- My system supports a "collect" method which removes a resource who's ref-count is 1 (which means that only the manager holds a reference). I can also force removal if I wish, but because handle is ref-counted, the ResourceData instance won't be destroyed until the last outstanding handle is deleted or goes out of scope, so there is no risk of deleting a resource that is still being used.

Loaders -- Loading resources form their persistent store is yet another separate responsibility. In my system, I specify the loader as a parameter to the manager method which acquires resources -- this is necessary since any call to acquire a resource may need to load that resource if it does not exist. If the requested resource does not yet exist, a new entry in the managers table (std::map) is created, and the loader is called through the supplied function pointer. This also means that, as long as there is enough memory, a resource will exist once this call is complete. To look for a resource that already exists (and without creating it if it does not) another member function, "find" is provided.


I'm very happy with my system and how it all fits together. It is completely generic (it can be applied to any Resource and ResourceInfo), separates loading into a separate code entity, makes no requirement on the Resource class (for example, resource classes are not required to support any interface, or derive from any base class) and is based entirely on standard library features -- which goes a long way to making it rock-solid and bug free. It even supports polymorphic resources -- that is, if the manager is templated against a base resource type, then it may manage any types derived from that base -- giving the client code tons of freedom.

throw table_exception("(? ???)? ? ???");

My resource management is very similar to Ravyne's in organization. I couple the Loader with the ResourceInfo, but it might be better if they weren't.

This topic is closed to new replies.

Advertisement