Thanks, very useful information.

2) The build system. Just like you do for your code, you create a type of "makefile" for your assets, often for a custom built build-system, but possibly also for an off-the-shelf one (e.g. XNA uses msbuild). Whenever you've updated some assets (e.g. saved/edited them, or updated SVN, etc) you run your build-script, which uses timestamps/hashes etc to only recompile the necessary files. The game doesn't contain any parsing/compilation code (just simple binary loading code); all the parsing/compilation code is pulled out into separate tools that are used by the build system.If you want to support modding, you can also ship the build system.

This seems very straightforward, and it will almost certainly make disk IO the loading bottleneck - which I can handle with overlapped reads.

3) The persistent build system. As above, but the build system sits in your system tray all day and watches your source asset directory. Whenever you change a file, it automatically recompiles it in the background. If the game is running, then when it's complete, it sends the game a message telling it to reload the modified file.

This seems very fancy Is there a preferred way of the directory watcher communicating with the game? My first thoughts would be some kind of shared Win32 event, or a named pipe. Or perhaps the game listening locally on UDP?

Thanks again.

OK - I've made a couple of changes to make my resource system work with asynchronous loading:


Instead of the cache returning [font=courier new,courier,monospace]T*[/font] it can now return a [font=courier new,courier,monospace]ResourceHandle<T>*[/font], which tracks whether the resource is loaded or not. The cache owns a map of these handles.

When a resource is requested and the cache doesn't have it, it calls [font=courier new,courier,monospace]ResourceLoader<T>::preLoad()[/font]. This can issue the asynchronous read and return a 'not loaded' handle, which is stored by the cache in a separate pending list and returned to the client immediately.

When [font=courier new,courier,monospace]ResourceCache<T>::update()[/font] is called (once per frame), it checks the list to see which asynchronous reads have completed, and calls [font=courier new,courier,monospace]ResourceLoader<T>::postLoad()[/font] for those. That's where the GPU objects are created, for example. The resource is removed from the pending list, and it's flag is set to loaded.

So the overall flow will be something like this:

LoadLevel
Request all resource from the cache, which returns the handles.
While any resources are still being loaded:
Loop, draw the loading screen and call ResourceCache::update().


I think that should work. Seems reasonable?

I did this, and it was very fast and easy to implement, but artists got annoyed when trying to roll back older versions of assets and the cache wouldn't update itself (because the timestamp of the old file was older than the timestamp of the cached file). Hashing is definitely the way to go. (You can always do both and give the ability to turn off hashing to speed up loading)

Probably stating the obvious, but you could just check the timestamps for equality.

I did this, and it was very fast and easy to implement, but artists got annoyed when trying to roll back older versions of assets and the cache wouldn't update itself (because the timestamp of the old file was older than the timestamp of the cached file). Hashing is definitely the way to go. (You can always do both and give the ability to turn off hashing to speed up loading)

I never check for a newer timestamp, just a different timestamp. What you mentioned is one reason, the other reason is that I've run into applications that like to change the timestamp manually and it's not always tuned to the system clock for some reason, so newer files can have an older timestamp.

You can then do a hash check if the timestamps are different to make sure the files are actually different before recompiling them.

Using hashes is probably fine if you're only doing it on load, though it will be a bit slower it shouldn't be too bad. But if you want to do live updating then it's a killer if you have a large resource set.

I've seen hashing schemes that left the actual data processing step in the dust, meaning they effectively have zero impact on running time.

Most hashing is faster than data processing, we're talking about files that are pre-processed and thus don't require any runtime processing for the most part. (Except for possibly textures.) No matter how careful the implementation, hashing will have an effect on load times in this case. For live updating, the difference between checking a timestamp and reading and hashing a file that otherwise doesn't need to be read at all is obviously huge when being done continuously in the background alongside your game loop.

Thanks everyone, this is great advice

I've almost got the basic async loader working, but I've run into a problem.

Some of my resources have dependencies on others, such as a material being dependent on a shader program. In my original system I was handling these dependencies like so:


IResourceLocator* locator = new ArchiveResourceLocator("resources.zip");

ResourceLoader<Shader> shaderLoader = new ResourceLoader<Shader>(locator);
ResourceCache<Shader> shaderCache = new ResourceCache<Shader>(&shaderLoader);

// The material loader needs to be able to resolve shader references...
ResourceLoader<Material> materialLoader = new ResourceLoader<Material>(locator, &shaderCache);
ResourceCache<Material> materialCache = new ResourceCache<Material>(&materialLoader);

// A request for a material will now 'automagically' use the shader cache...



This is a problematic when the resources are loaded asynchronously, as a request for a material also has to wait for the shader dependency to be loaded.

When an asynchronous read has completed and the resource is ready to be 'loaded' for real, I will need to check whether it's dependencies are fully loaded before proceeding. I can see this "resources waiting for resources" turning into a big mess quite easily.

I don't think I have any other choice, but I was hoping to see what others think before I go for it.

Thanks again