Resource Cache and Models

Started by
4 comments, last by SeanMiddleditch 9 years, 11 months ago

Hello,

I want to discuss loading composite resources and find a good way to loading them. I'm using my own file format for storing other files(like zip). I'm using my own binary format for loading 3D models too. My target is to load raw data, then initialize them(creating textures, xml documents, shaders etc.) and then release them. But there is a problem, if models come before textures. I'm having two solutions now:

1) Organizing files to load textures before models

2) Load model, then give a request for texture and if texture is loaded, then send it to every model which want it

But, isn't here a better way(I want to support multithreaded programming)?

It's a good way to loading resources asynchronously?

Advertisement
I load resources in groups. Loading a resource has 3 steps, (1) streaming in the raw data, (2) parsing/initializing it, and (3) resolving links between resources.
Step 1 is handled by the file system and 2/3 by the specific factories. As soon as the data is streamed in, the factory performs step 2. During step 2, you can request more resources to be loaded (such as your textures) -- if you do this, they're added to the same "loading group". Only once every resource in a group has completed steps 1&2, then step 3 is performed for each resource in the group.

I've implemented 2 ways for handling this. The first way is like what Hodgman has mentioned above: It works when references to other resources are found during the interpretation step. The other way is that of bundles. An archive file has a table of content for each single stored resource as usual, but the table of content refers to so-called load units instead of resources directly. A load unit is referred to by just a file offset and byte count and denotes the section of the file that has to be loaded when the corresponding resource is requested. So a load unit may store a single resource, but it may be also store a sequence of resources (the said bundle). Because all entries in the table of content which denote one resource of a bundle show the same load unit, requesting such a resource causes the entire sequence of resources to be loaded. So load units are loaded (and unloaded) ever in their entirety.

You probably want to be carefull in how you construct your load units, so that they load only what you need got that unit. Dependency checks in your bundle build step can alleviate these issues, so that you only bundle resources that are often used together. This will remove your problems in that the load unit has to be loaded completely.

Worked on titles: CMR:DiRT2, DiRT 3, DiRT: Showdown, GRID 2, theHunter, theHunter: Primal, Mad Max, Watch Dogs: Legion


You probably want to be carefull in how you construct your load units, so that they load only what you need got that unit. Dependency checks in your bundle build step can alleviate these issues, so that you only bundle resources that are often used together. This will remove your problems in that the load unit has to be loaded completely.

You are right, except that loading bundles as an entirety is not a problem but a feature. Its sense is to reduce mass storage accesses (assuming that not all platforms have an SSD). A load unit may of course contain just a single resource, but it may also (as an example) contain the mesh plus necessary textures plus a skeleton plus animations for a model, however you want the toolchain bundle the resources.

My target is to load raw data, then initialize them(creating textures, xml documents, shaders etc.) and then release them. But there is a problem, if models come before textures. I'm having two solutions now:
1) Organizing files to load textures before models
2) Load model, then give a request for texture and if texture is loaded, then send it to every model which want it
But, isn't here a better way(I want to support multithreaded programming)?
It's a good way to loading resources asynchronously?


One option is to allow request handles, so resource A can refer to as-yet-unloaded resource B. When you want to load a texture, you immediately create a new engine-side Texture object and assign it to the model's member variable as appropriate. This handle doesn't actually yet refer to a real texture; it either refers to some kind of empty texture or a placeholder missing texture. The texture can load in the background. Objects with dependencies should keep track of how many of their dependencies are still pending and leave themselves in a pending stuff until the dependencies are resolved.

I've seen approaches that use multiple layers of indirection, so e.g. you are given a pointer to a RequestHandle which in turn points to a Resource (so you have to m_MyResource->GetResource()->Foo() to interact with the resource). I dislike this approach, though it certainly works (even on Very Large Games). I prefer allowing resources to be streamed in such that every request for a resource directly and immediately returns an instance of that specific resource type, which just also handles state and dependent loads (possibly via a Resource base class). Either approach works just fine with threads, though it's definitely easier in the first approach.

A further advantage of this system is that you _can_ allow use of the handles before they're fully loaded. This can represent Level of Detail objects (you're not done streaming in the whole resource, but a portion of it is ready; see how Unreal textures stream in, for instance). It also makes it easier during development to deal with missing resources; just have the handle refer to the "missing" default instance of the resource type until-and-if the load completes. You may find during early iterations you want to get into game as quickly as possible to test various features and don't care whether certain resources load immediately or at all, while of course in release builds you want every resource to be ready before you exit any loading screens. Loading groups, pending states and dependent loads, and post-load callbacks all make it easy to deal with when and if you need to. If resource A needs data from resource B, a post-load step allows you to fix up data once all the dependent loads are complete.

You can build bundles on top of this system, of course, as well as all other kinds of pre-loading and prioritization systems. The above isn't a full solution, but it has some advantages (especially in the short-mid term of development). It also plays nicely with loose assets, which is handy either during early development or if you plan to allow easy and ubiquitous modding (like Bethesda's RPGs, for instance).

Sean Middleditch – Game Systems Engineer – Join my team!

This topic is closed to new replies.

Advertisement