Writing classes for a generic asset manager?

Started by
5 comments, last by rozz666 10 years, 10 months ago

TL;DR:

Comments:

AssetManager

*assetcache (identifier -> assetID map, assetID -> asset map)

GeneralAssetManager (identifier(filepath) -> fromcache/fromhdd -> assetID with refcounter to raw uninterpreted data)

ShaderAssetManager (identifier(assetID to raw data) -> fromcache/fromGeneralAssetManager -> shader object ID with refcounter)

ImageAssetManager (identifier(assetID to raw data) -> fromcache/fromGeneralAssetManager -> decompressed image data ID with refcounter)

TextureAssetManager

...

I need some classes to allow me to implement asset managers.

I expect them to be reusable template classes.

I dont want a do-it-all manager that handles everything from textures to xmls.

Managers i think ill need:

-General asset manager

*gateway between external sources (hdd, network?) and RAM

*loads the raw resource data into RAM

-GPU asset managers

*gateway between RAM and video memory

*one for shaders, another for textures...

*interprets raw data, uploads to GPU, creates the object to be used in code

For example, if i have a picture on hdd, ill tell the general asset manager to load it using a raw resource identifier (the filepath). It then checks its cache and if its not there it loads it. It will return an integer identifier (like a hash) unique to each loaded resource that i can use to refer to the loaded asset. The integer is random but shouldnt overlap with other assets.

Now i have my identifier.

Then i will ask lets say the texture cache to give me the GPU-side texture. I give it the raw asset identifier, it checks its identifier-textureobject cache, interprets and uploads raw data if necessary, and then hands me the texture object.

Ideally, i would preload the assets in memory and store the identifiers somewhere (if i have hardcoded assets ill need) and then interpret and upload those when necessary. I would only access resources through the filepath and general asset manager if i get the filepath dynamically. If i already know the filepath ill preload it at some point and set the resulting identifier in a hardcoded variable.

For textures, i might want to decompress the image but not yet upload to GPU, so i could use a cpu side texture manager that simply interprets (decompresses) the raw asset.

And then some tools for handling the lifetime of objects. The resource identifiers (which you get from the managers) should probably have a reference counter (ref. count managed along with the loaded resource). The refcount would only prevent deletion of assets in use, deleting assets that are not referenced would probably be manual.

Anything to improve on? Does this already seem fairly solid? As mentioned ill need reference counts to the asset identifiers i use to access already loaded assets, how should i implement that (the refcount will reside with the loaded asset)?

o3o

Advertisement

Whenever someone write the words "Generic <INSERT BLANK HERE> Manager", a little part of me dies...

What concrete problem are you trying to solve here? This looks like a hideously over-engineered approach, but it's hard to tell unless you provide a concrete use-case.

For example:

- Are you trying to solve the issue of low-latency loading from optical media?

- Are you dealing with a low-memory embedded device, and trying to minimise memory footprint?

- Do you have a multi-gigabyte world that needs to be progressively streamed to the GPU?

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

My goal is to prevent loading multiple instances of a resource in memory.

The 'generic' resource manager doesnt handle multiple types of objects. It only handles loading the raw data from disk, the actual type related stuff is done by specialized managers.

The 'generic' resource manager could be said to be just a cache for the hdd.

The above is enough to deal with:

-Memory use (each resource loaded to ram only once, not referenced raw files can be unloaded on command...)

-Latency (i can tell assets to be loaded at the start and theyll stay in the cache, with some metadata for the refcount system if needed)

However, raw data is usually just kept in until parsed/decompressed so it doesnt really address your third point.

For that i need things to manage the lifetime and uniqueness of textures, shaders and other GPU objects.

Lets say that i identify textures using their filenames. When something in the game requests a texture given a name (assuming the name is not known from the start - in which case the texture could be loaded before its needed), the texture manager cache will be checked first and if its not there itll be loaded using the file manager/image manager (generic manager=file manager)

This will be used mostly for nonchanging assets that can be assigned an identifier (their file name for example, or something defined in some text file linking it to an actual file)

I dont intend to implement more than the file manager/cache now though.

o3o

That sounds like a suitable job for a HashMap. Not clear why you need a whole set of classes just to enforce uniqueness?

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

The manager/cache will also have reference counting built in + the code to load things not yet in the cache when needed.

Im using unordered_map (AFAIK this is practically hashmap) internally though, intellisense not working with it though gotta make a thread.

o3o

As above, I'd avoid the 'catch all' words 'generic' and 'manager' if you're trying not to write 'do all' classes wink.png

It sounds like a silly nit-pick, but actually breaking down each of the components to discover it's one true purpose is very useful in coming up with a clean design. "Manager" classes usually don't have a single clear purpose -- like performing caching, lifetime management and filesystem interactions...

My "generic manager" is called a "blob loader" - it loads blobs of bytes into RAM.
My "foo managers" are "foo factories", because they're an incarnation of the factory pattern that turns bytes into objects.
When trying to load an asset, I pass the name (a hashed ID), the loader and the factory to an "asset map", which first checks if it already exists, and if not, sends a request to the loader to fetch the bytes and forward them on to the factory, which in turn stores the constructed object in the map.

If reference counting is needed, you can use boost::shared_ptr and boost::weak_ptr. E.g.


template <typename Resource, typename Id>
boost::shared_ptr<Resource> CachedResourceLoader::load(const Id& id)
{
    if (boost::shared_ptr<Resource> res = cache->get(id))
        return res;
    boost::shared_ptr<Resource> res = loader->load(id, boost::bind(&Cache::notifyRelease, cache, id)));
    cache->put(res, id);
    return res;
}

template <typename Resource, typename Id>
boost::shared_ptr<Resource> UsedResourcesCache::get(const Id& id)
{
    Elements::iterator it = elements.find(id); // elements: boost::unordered_map<Id, boost::weak_ptr<Resource> >;
    if (it == elements.end())
        return boost::shared_ptr<Resource>();
    return elements->lock();
}

template <typename Resource, typename Id>
void PermanentCache::notifyDestroyed(const Id& id)
{
    elements.erase(id);
}

boost::shared_ptr<Image> ImageLoader::load(const std::string& filename, boost::function<void()> onDestroy)
{
    return boost::shared_ptr<Image>(new Image(filename), boost::bind(&ImageLoader::imageDeleter, onDestroy, _1));
}

/*static*/ void ImageLoader::imageDeleter(boost::function<void()> onDestroy, Image *img)
{
    delete img;
    onDestroy();
}

When last reference to a resource dies, imageDeleter will be called, which will notify the cache to remove an empty weak_ptr.

This will also work with std version of shared_ptr, function, etc. if your compiler supports it.

This topic is closed to new replies.

Advertisement