Asynchronus Resource Manager

You may want to read the thread

https://www.gamedev.net/topic/688440-loading-file-paths-and-how-to-access-them/

in the neighbouring forum

You may want to read the thread

https://www.gamedev.net/topic/688440-loading-file-paths-and-how-to-access-them/

in the neighbouring forum

Thats interesting, thanks! Didn't realise it was a similar topic (because of the title).

The approach that I have listed is similar to what Hodgman posted (#12).

As well as being asynchronous, I wanted to support:
* loading from compressed archives.
* encryption / decryption.
* streaming into memory allocated by the user. E.g. On some platforms, the user may have a block of GPU memory and they want the asset system to stream a texture file directly into the GPU.

So I use multiple callbacks - when the uncompressed file size is know, the first callback lets the user allocate memory for the asset however they like. A second callback lets them know that streaming has completed.
Internally the system may be streaming directly into the users buffer, or streaming compressed blocks into its own internal buffer, and then decompressing into the users buffer.

This is pretty much what I want to support.

I looked through your post on the other thread and have a few questions.

Do you run a single thread for this system or is it part of a larger thread pool?

How do you handle the different file types (archives, etc)?

Would your system work something similar to the following:

//MAIN THREAD
ResourceHandle<T> handle;
pushAsyncRequest(handle, filepath, factory);
return handle;

//THREAD 1
size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(allocSize);
FileStream f = fs.open(filepath);
MemoryStream mem(f); //Loads all the data from the file into memory

//MAIN THREAD - Callback from thread 1
factory.onCompletion(mem); //Deserializes the data to consturct the resource
//Resource is now ready to use & handle is populated 

Would your system work something similar to the following

Yeah pretty much like that, except in your version there's no point in having the allocate callback as you don't make use of the allocated memory in anyway. You read the file into your own in-memory buffer and then pass your buffer to the factory in the completion callback.
What I do is stream the file directly into the factory's allocation, and then notify the factory once this operation is complete.

//THREAD 1
size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(handle, allocSize);
FileStream f = fs.open(filepath);
f.read(data, allocSize); //Loads all the data from the file into memory

//MAIN THREAD - Callback from thread 1
factory.onCompletion(handle, data); //Deserializes the data to construct the resource
//Resource is now ready to use & handle is populated 

I try to structure most of my formats so that there is minimal deserialization required, and most of them deserialize in-place without extra allocations.
When in-place deserialization isn't possible, then the memory allocated in factory.allocate will be released during factory.onCompletion.

If you are using a Material factory to load a material, wont all materials be the same size? What does the measuring stage actually do because wouldn't you just need to allocate a new Material then read the uncompressed data into it?

Each material may use a different shader, and each shader may declare a different amount of uniform variables and textures. That means that different materials can greatly differ in size :)
Off topic but FWIW, I don't store individual materials as assets, instead I have a "material pack" asset. Each model file will likely contain many materials (for different parts of the model), so when importing a model from an artist, it generates a material-pack asset to go alongside that model's geometry asset.

Do you run a single thread for this system or is it part of a larger thread pool?

Internally on Windows I use the native async file system API (see the lpOverlapped parameter of ReadFileEx)... however, this API still sometimes blocks on occasion (I think when you have too many async operations in flight), so whenever I need to call one of those functions I push a job to a background thread / thread pool so that the main thread(s) won't stall. The main thread(s) poll the blob loader once per frame to check if any file streaming operations have completed, and if so the completion callbacks are triggered (so these happen on the main threads). However, I limit the number of callbacks that can be called per frame, again to avoid stalling the main threads, as certain deserialization operations can be expensive. I plan to develop a system where each factory can provide a hint of how expensive it's callbacks are to help the blob loader manage this.

How do you handle the different file types (archives, etc)?

When I create a blob loader, I pass it an enum value of which kind of implementation I'd like it to create internally (local loose files, packed archive file, remote/network file system).

how does your system take the initial sys.m_materialFactory.Load call and output a Material

I have something similar to your ResourceHandle, but mine's just called Asset and is basically a union{void* ptr; ptrdiff_t id;};
The different asset types (T) implement wrappers around this common handle that let you gain access to the deserialized type in their own way. e.g. my renderer's actual texture type is TextureId, so the asset version of a texture is called TextureAsset and it has a GetId member function that returns a TextureId (this is equivalent to having ResourceHandle). It's only safe to call GetId once the texture has finished loading though, so there's two mechanisms to determine that.
Instead of allowing you to query Assets (TexureAssets/etc) as to whether they're finished loading or not, I load all assets in batches and associate them with this batch-managing object, called an AssetScope. When loading an object, you need a factory, an asset-scope, an asset-name to load, and a blob-loader. Once you've finished adding load requests into the batch, you can call Close on the AssetScope to tell it that no more assets will be added to this batch.
The first way to tell when the AssetScope is complete is to periodically poll it:

struct Example
{
	AssetScope m_assets;
	TextureAsset* m_foo;
	TextureAsset* m_bar;
	bool m_loaded;
	Example(BlobLoader& loader, TextureFactory& textureFactory)
		: m_loaded()
	{
		AssetName nameFoo("foo.tex");
		AssetName nameBar("foo.tex");
		m_foo = textureFactory.Load( nameFoo, m_assets, loader );
		m_bar = textureFactory.Load( nameBar, m_assets, loader );
		m_assets.Close();
	}
	void Update()
	{
		if( m_assets.Update() == AssetScope::Loaded )
			m_loaded = true;
	}
	bool Loaded() const { return m_loaded; }
	void Draw()
	{
		if( !m_loaded )
			return;
		TextureId foo = m_foo->GetId();
		DrawSomething(foo);
	}
};

And the other is to register a call-back before closing the asset-scope:

struct Example
{
	AssetScope m_assets;
	TextureAsset* m_foo;
	TextureAsset* m_bar;
	bool m_loaded;

	Example(BlobLoader& loader, TextureFactory& textureFactory)
		: m_loaded()
	{
		m_foo = textureFactory.Load( AssetName("foo.tex"), m_assets, loader );
		m_bar = textureFactory.Load( AssetName("foo.tex"), m_assets, loader );
		m_assets.OnLoaded( [this](){ m_loaded = true }, Task::CurrentThread() ); // code to run on completion, and which thread to run it on
		m_assets.Close();
	}

	bool Loaded() const { return m_loaded; }

	void Draw()
	{
		if( !m_loaded )
			return;
		TextureId foo = m_foo->GetId();
		DrawSomething(foo);
	}
};

Would your system work something similar to the following

Yeah pretty much like that, except in your version there's no point in having the allocate callback as you don't make use of the allocated memory in anyway. You read the file into your own in-memory buffer and then pass your buffer to the factory in the completion callback.
What I do is stream the file directly into the factory's allocation, and then notify the factory once this operation is complete.

//THREAD 1
size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(handle, allocSize);
FileStream f = fs.open(filepath);
f.read(data, allocSize); //Loads all the data from the file into memory

//MAIN THREAD - Callback from thread 1
factory.onCompletion(handle, data); //Deserializes the data to construct the resource
//Resource is now ready to use & handle is populated 

I try to structure most of my formats so that there is minimal deserialization required, and most of them deserialize in-place without extra allocations.
When in-place deserialization isn't possible, then the memory allocated in factory.allocate will be released during factory.onCompletion.

If you are using a Material factory to load a material, wont all materials be the same size? What does the measuring stage actually do because wouldn't you just need to allocate a new Material then read the uncompressed data into it?

Each material may use a different shader, and each shader may declare a different amount of uniform variables and textures. That means that different materials can greatly differ in size :)
Off topic but FWIW, I don't store individual materials as assets, instead I have a "material pack" asset. Each model file will likely contain many materials (for different parts of the model), so when importing a model from an artist, it generates a material-pack asset to go alongside that model's geometry asset.

Do you run a single thread for this system or is it part of a larger thread pool?

Internally on Windows I use the native async file system API (see the lpOverlapped parameter of ReadFileEx)... however, this API still sometimes blocks on occasion (I think when you have too many async operations in flight), so whenever I need to call one of those functions I push a job to a background thread / thread pool so that the main thread(s) won't stall. The main thread(s) poll the blob loader once per frame to check if any file streaming operations have completed, and if so the completion callbacks are triggered (so these happen on the main threads). However, I limit the number of callbacks that can be called per frame, again to avoid stalling the main threads, as certain deserialization operations can be expensive. I plan to develop a system where each factory can provide a hint of how expensive it's callbacks are to help the blob loader manage this.

How do you handle the different file types (archives, etc)?

When I create a blob loader, I pass it an enum value of which kind of implementation I'd like it to create internally (local loose files, packed archive file, remote/network file system).

how does your system take the initial sys.m_materialFactory.Load call and output a Material

I have something similar to your ResourceHandle, but mine's just called Asset and is basically a union{void* ptr; ptrdiff_t id;};
The different asset types (T) implement wrappers around this common handle that let you gain access to the deserialized type in their own way. e.g. my renderer's actual texture type is TextureId, so the asset version of a texture is called TextureAsset and it has a GetId member function that returns a TextureId (this is equivalent to having ResourceHandle). It's only safe to call GetId once the texture has finished loading though, so there's two mechanisms to determine that.
Instead of allowing you to query Assets (TexureAssets/etc) as to whether they're finished loading or not, I load all assets in batches and associate them with this batch-managing object, called an AssetScope. When loading an object, you need a factory, an asset-scope, an asset-name to load, and a blob-loader. Once you've finished adding load requests into the batch, you can call Close on the AssetScope to tell it that no more assets will be added to this batch.
The first way to tell when the AssetScope is complete is to periodically poll it:

struct Example
{
	AssetScope m_assets;
	TextureAsset* m_foo;
	TextureAsset* m_bar;
	bool m_loaded;
	Example(BlobLoader& loader, TextureFactory& textureFactory)
		: m_loaded()
	{
		AssetName nameFoo("foo.tex");
		AssetName nameBar("foo.tex");
		m_foo = textureFactory.Load( nameFoo, m_assets, loader );
		m_bar = textureFactory.Load( nameBar, m_assets, loader );
		m_assets.Close();
	}
	void Update()
	{
		if( m_assets.Update() == AssetScope::Loaded )
			m_loaded = true;
	}
	bool Loaded() const { return m_loaded; }
	void Draw()
	{
		if( !m_loaded )
			return;
		TextureId foo = m_foo->GetId();
		DrawSomething(foo);
	}
};

And the other is to register a call-back before closing the asset-scope:

struct Example
{
	AssetScope m_assets;
	TextureAsset* m_foo;
	TextureAsset* m_bar;
	bool m_loaded;

	Example(BlobLoader& loader, TextureFactory& textureFactory)
		: m_loaded()
	{
		m_foo = textureFactory.Load( AssetName("foo.tex"), m_assets, loader );
		m_bar = textureFactory.Load( AssetName("foo.tex"), m_assets, loader );
		m_assets.OnLoaded( [this](){ m_loaded = true }, Task::CurrentThread() ); // code to run on completion, and which thread to run it on
		m_assets.Close();
	}

	bool Loaded() const { return m_loaded; }

	void Draw()
	{
		if( !m_loaded )
			return;
		TextureId foo = m_foo->GetId();
		DrawSomething(foo);
	}
};

Fantastic!

This explains pretty much everything I needed to know, thanks for taking the time to write it :D

If I have any more questions, I'll post them here :P

Thanks @Hodgman