Jump to content
  • Advertisement
Sign in to follow this  
Althar

Resource Management - Candidates for "Resource"

This topic is 1530 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey all,

 

I was hoping I could get your opinion and views on the following question regarding Resource/Game asset management:

 

What constitutes a good candidate for a "Resource" in a game engine's Resource/Asset manager, in your opinion?

 

I have been tinkering with a Resource Management system for my game engine and trying to get my head around what to consider and what not to consider as a Resource. So far, I have :

  • Resource : a basic class with an ID (string/hash) and an origin (runtime or disk)
  • ResourceHandle : a basic wrapper around a Resource - essentially a shared pointer, with some additional functionality (proxy-like with optional hot-swapping/live-reload of the underlying resource)
  • ResourceManager : some people might think of a better name for this class - essentially a class with a map of weak references to Resource objects with functions to Load/Save ResourceHandle objects (note the manager does not handle loading/saving, it merely dispatches these commands to the appropriate ResourceLoader and ResourceWriter which can be implemented as required for the various Resource types. So far I have only one instance of the ResourceManager but could quite easily create more (ie. per package, bundle, level, etc.)

 

So far, I have basic disk-based resource types such as Text, Binary which I can load and save - the bit that I am slightly torn with, is whether to consider graphical resources as Resource objects as well (meaning I would introduce strong coupling between the Rendering side and Resource management side) - This is the way I remember XNA doing it (possibly Mono as well?) and syntactically it is very user friendly and easy to use - on the other hand it does imply building a good section of the rendering library on top of the resource system - do you reckon this dependency is justified? If not, how would you approach it?

 

Alternatively, I could keep managing them separately, with the optional compound object (ie. a Model could be a resource containing a Mesh, Textures, etc.) but it does introduce heavier syntax requirements to store the graphical objects, and makes it less user-friendly : I would also end up in a situation where I would have a Texture2D (Rendering-side) and a Texture2DResource (Resource-side) which sort of seems repetitive.

 

Below is a sample of my approach in its current state - the shader code is loaded from a text file and preprocessed and is then fed to a ShaderProgram which has no relationships with the Resource system :

// Setup shader program
{
      const gfx::ShaderSourceHandle vertexShaderSource   = resourceManager.load<gfx::ShaderResource>("data/effects/debug/vs_debug.vsh");
      const gfx::ShaderSourceHandle fragmentShaderSource = resourceManager.load<gfx::ShaderResource>("data/effects/debug/fs_debug.fsh");

      m_shaderProgram = m_renderInterface.createShaderProgram( vertexShaderSource->getSource().c_str(), fragmentShaderSource->getSource().c_str() );
      m_shaderProgramToken.m_shaderProgram = m_shaderProgram;
} 

The problem arises when I think about ways of sharing this shader around - I could have a map of shaders or else, but wouldn't it just be duplicating what the Resource management system is designed for? Additionally, what happens if I wanted to enable live-editing of the shader : the user would make changes to the shader source files, but then the ShaderProgram has no notion of the Resource and by the time it has been constructed, has lost all dependencies with the shader source.

 

What are you thoughts? Please feel free to criticise my approach and share with me a better alternative.

 

Thanks in advance!

Share this post


Link to post
Share on other sites
Advertisement


So far, I have basic disk-based resource types such as Text, Binary which I can load and save - the bit that I am slightly torn with, is whether to consider graphical resources as Resource objects as well (meaning I would introduce strong coupling between the Rendering side and Resource management side) ...

If the resource management isn't responsible for managing graphics as resources ... what part is responsible for it? Textures, sounds, meshes, animation clips, scripts, ... are all classical resource types. "Text" and "binary" as resource types ... what's that exactly?

 

The rendering side should be divided up into a higher level and a lower level. The higher level is responsible for traversing the scene objects and determining visibility of renderable objects. Perhaps it may trigger mesh generators like skinning or the like if it detects such a necessity. Scene objects determined as visible are requested for all stuff necessary to render them. Then graphic rendering jobs are generated and passed to the lower level of rendering. The lower level is not involved in resource management, the higher level may be (dependent on the implementation). This is okay.

Share this post


Link to post
Share on other sites

Thanks for your feedback, it is very much appreciated biggrin.png .

 

I should have probably been more specific : the Text and Binary resource types were just examples of base classes used to implement loading and saving of "generic" text/binary files - obviously, this can be extended/derived for handling more specific resources (e.g. config files, raw data, etc.). These base classes hide the platform-specifics of loading data off the disk but you are free to then interpret the data in any way you wish.

 

As it stands, I do already have a concept of low and high level renderer : ie. the hardware abstraction layer / "render interface" that wraps your typical textures, buffers, shaders and commands - and then a flexible token-based pipeline that executes pre-compiled rendering streams built as a result of traversing the scene/pvs, etc.

 

The fine line which I was trying to break was whether a Texture, as created by my "render interface" (which internally holds the API-specific texture, states, etc.) should be considered a Resource or whether it is preferable to defer resource management to a higher level (e.g. a Material instance might hold several Textures). I have seen it done both ways so there is no right or wrong, but I would be interested to know what are people's experiences with any one approach and what sort of challenges they had to deal with later down the line.

 

Hopefully, this will make sense?

Share this post


Link to post
Share on other sites
I usually find it useful to think of resources in three phases:

- Loading from disk/network/etc.
- Memory management
- Consumption

These should be three totally distinct and self-contained aspects of your code. Acquiring a resource from disk (or any other source) should be modular so that you can acquire things which are not "resources" in the same way, using the exact same code path. Similarly, memory management is a different job than loading data. Generally this is the more sophisticated layer in game engines, because you need to do things like enforce asset budgets, swap out low-res alternative assets when memory is scarce, etc. But again, it's a totally self-contained and agnostic concept - you can use memory management on your internal data structures just as well as you can on a texture or audio clip.

The final layer is consuming the memory-managed resource. Ideally this should be as simple as telling the memory handler "hey, I want that piece of data" and then the data sticks around for you to use in whatever way necessary until you say "ok I'm done."

By keeping these layers strictly separate, you end up with a much nicer division of labor and avoid the trap of having resource god objects that contain half of your engine.

Share this post


Link to post
Share on other sites

You're using some words that show you might be violating the single responsibility principle.

 

Both "resource" and "manager" are vague.  What exactly is a resource? What is a manager?  When I see them in code it usually means the developer didn't fully design the system. An exception to that rule would be an enumeration of all the different types of assets, which might be called 'resource', essentially a numeric constant that tells you if the thing you are indirectly referencing is a model, or texture, or animation, or sound, or state machine, or map, or spawn point, or some other kind of thing.

 

"Resources" is a generic term, and could be textures, models, sprites, animation clips, audio clips, level maps, and much more.

 

"Manager" is a very generic term that usually indicates a combination of loader/unloader, cache, and proxies that serve until the asset is loaded. Are they streamed from the network, or from disk, or from compressed archives? Are they loaded and cached, and for how long? Are they ever invalidated or discarded? Every answer suggests a different name for a subsystem.

 
It usually doesn't make much sense to intermix your assets.  They often have different lifetimes. Some may be stored in specialized, dedicated memory or external hardware, like storage in the GPU. They often have different usage patterns. They often  have different sizes, different alignment constraints, different traversal patterns. Games tend to have a pool of textures, a pool of audio, a pool of models, a pool of whatever else. They tend to 'manage' them by having various states of unloaded, partially loaded, and fully loaded.

 

It usually is pretty simple to build up some generic cache classes, generic streaming classes, generic proxy classes, and then specialize them.  Then you might have specialized classes for a TextureProxy, TextureCache, TextureStream. You might also have a specialized set of classes for ModelProxy, ModelCache, ModelStream. And an AnimationProxy, AnimationCache, and AnimationStream family. Etc.

 

edit: In case you aren't familiar with them, Proxy objects allow the code to return immediately rather than blocking until the stream is complete. You can use the proxy immediately, with the caveat that it might contain a simple placeholder rather than the final asset. Depending on how you build your engine you might allow them to be automatically swapped out, you might have an event that is broadcast when assets are loaded, you might have your system query the proxy to see a state of unloaded, partially loaded, or fully loaded, you might have something else entirely. Many game consoles through history have implemented this in their hardware, you could tell the device to load from a block of the cartridge and immediately use the resource, the hardware would swap it in place when the streaming is complete.

Edited by frob

Share this post


Link to post
Share on other sites
I usually find any kind of shared_ptr-like use of resources to just break down at all. There's various management and perf issues that just don't work all that well.

Far more preferable to use a POD handle type (i.e. an integer). You can pack the integer so that it's both an index for fast O(1) worst-case lookup in the resource manager and to also have a version tag so indices can be recycled safely.

You can pass around this integer. You can reload the underlying resource without invalidating the integer. You can sort by the integer. You can efficiently shove it into other POD data structures and memcpy things around.

Share this post


Link to post
Share on other sites

Some interesting points, thanks guys!

 

I admit that I am guilty of using the "Resource Manager" terminology like many others, but quite frankly I cannot think of an other name for it off the top of my head - I am open to ideas ?

 

I purposefully did not discuss caching, streaming or memory management in general as I did not believe it was relevant/a factor to my original question - I fully agree with ApochPiQ that there are several abstractions before even entering the realm on resource management aka : "as a user of this code-base, how do I get this textured model into my rainbow level?

 

It is this aspect of resource management which I am trying to get my head around - the frontend, from a naive user's perspective : what is a "Resource" and how can I load one? 

The end user doesn't want to have to worry each time about what cache and what policy to use, where the resource is going to get loaded in memory, etc. The end user wants to be able to request something and use it as quickly as possible.

 

Just like you say frob, the use of the word resource/asset tends to be very vague (hence my original question). I find it interesting that there is a recurring pattern in games, where Textures are generally considered as "assets", whilst Vertex/Index Buffers are not : fundamentally, both are graphical resources in that they present some form of prolonged persistence during the lifetime of the program and are expected to drive the simulation/visuals it in some way.

 

What is the acceptable boundary for a resource in that case, if there is any at all after all? And, also, how do you encapsulate/hide those layers of complexity to provide the end-user with a friendly interface, which does not imply violating every programming law out there?

Share this post


Link to post
Share on other sites

What is the acceptable boundary for a resource in that case, if there is any at all after all? And, also, how do you encapsulate/hide those layers of complexity to provide the end-user with a friendly interface, which does not imply violating every programming law out there?

Data structures and algorithms are tightly coupled.  There are some old and famous quotes I won't look up but will paraphrase:  You show me your algorithms and I'll describe the data structures. You show me your data structures and I can describe your algorithms.

 

Exactly how you define your data boundaries will strongly depend on your algorithms.  Exactly which algorithms you use will be heavily dependent on the data you store. Your performance will be directly related to how well the algorithms and data structures match the intended machines.

 

There are many general guidelines that are driven by today's hardware.  There were old guidelines driven by previous generations of hardware but are no longer applicable.  And there will be future hardware that will invalidate today's rules.

 

One facet of today's rules is that bulk data (like the long lists of vertex data and texture data and audio data, etc) should be organized for access in parallel sequential data streams processed together. Data locality is important, so ensure it is kept near the location it will be consumed; keep graphics data on the graphics card, keep audio data on the audio processor, keep CPU data in cache-friendly ways. That means a generic "resource manager" that keeps your graphics data in main memory is less than ideal, it is similarly bad if it keeps other data away from its intended process, or keeps data in a format that does not allow parallel sequential data streams.  

 

The player does not care about the game's internals.  They don't care if the game is using point clouds or triangle strips, or volumetric rendering, all they care is that it looks good.  They don't care if the sound track was recorded with a live orchestra in a specific country or generated by ProTools, they just care that it sounds good.  They don't care if your data structures are laid out as an array of structures or as a structure of arrays, they just care that the game is fast and responsive.

Share this post


Link to post
Share on other sites


I purposefully did not discuss caching, streaming or memory management in general as I did not believe it was relevant/a factor to my original question - I fully agree with ApochPiQ that there are several abstractions before even entering the realm on resource management aka : "as a user of this code-base, how do I get this textured model into my rainbow level?"

 

It is this aspect of resource management which I am trying to get my head around - the frontend, from a naive user's perspective : what is a "Resource" and how can I load one? 
The end user doesn't want to have to worry each time about what cache and what policy to use, where the resource is going to get loaded in memory, etc. The end user wants to be able to request something and use it as quickly as possible.

The user has an API with 2 routines (perhaps per type of resources) both of which return a resource requested by an identifier. The one routine works synchronously and the other asynchronously. They differ only in the case that the resource need actually be loaded from disk, in which case the synchronously working routine will block until the resource is available, while the asynchronously working routine will return a "need to look later" kind of result and the actual result is needed to be fetched later from a 3rd routine of the API. There is no need to consider the inner workings of the resource manager besides the question whether the user is willed to wait for the result.

 

Because the engine should be data driven, close to everything can be handled as a kind resource. Many of them are already enumerated in posts above.

 

Coming back to the declaration of "text" and "binary" resources ... that seems me not meaningful in the end. The resource manager (cache, loader) need not know about types of resources. Instead each resource is seen as a blob. The toolchain has generated the blobs and has stored it into the package file, usually with some metadata just suitable for reading and storing it. Only the instances that request resources from the manager need to know about the few types of resources they work with. And for them the distinction between "text" and "binary" is too weak.

Share this post


Link to post
Share on other sites

I only have asynchronous binary blobs in my "resource loader". The gist of it is something like:

//Strings don't belong in a game engine. Refer to files by a hash of the path.
struct AssetName
{
    explicit AssetName(const char* name);
    explicit AssetName(u32 nameHash);
    u32 nameHash;
};

//Loads files from disc. User provides callbacks for allocating (once file size is known) and completion (once loading is complete), which will be called asynchronously.
class BlobLoader
{
public:
        struct Request
        {
                typedef void* (*PfnAllocate)(u32 blobSize);
                typedef void  (*PfnComplete)(u8* blobData, u32 blobSize);
                PfnAllocate     pfnAllocate;
                PfnComplete     pfnComplete;
                void*           userData;
        };
        bool Load(const AssetName&, const Request&);
};

On top of this layer, I have "AssetScope" which is a basically a cache -- when requesting a resource, the cache will be checked before issuing a request to the BlobLoader.
On top of that, there's "BlobFactory"-derived classes, which make use of an AssetScope and a BlobLoader to load, cache and deserialize/init/deinit specific types of game resources.
The game programmer typically doesn't use the BlobLoader at all -- they mainly interact with the *Factories (passing the appropriate AssetScopes -- e.g. things that are needed during menus vs in-game go into different caches in my system).

Edited by Hodgman

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!