Jump to content
  • Advertisement
Sign in to follow this  
nickyc95

Asynchronus Resource Manager

This topic is 580 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi,

So I am building a resource manager for my engine and I have currently gotten stumped as to what approach to take with regards async behaviour.

I have an idea at the moment but i'm not sure if it would be a good approach. (I will just post the ideas for async as it will be very similar for non-async)

 

Approach

  • Request to the resource manager to load a file asynchronously 
  • Check the file isn't loaded / already loading 
  • The resource manager creates a new AsyncFileRequest with a callback to load the data from the file, this request is placed in a queue that is handled by the file manager. When the manager picks up the request, it will open the disk file (possibly loaded into a memory stream asynchronously)
  • Then calls the callback so that the data can be extracted from the file and the resource can "load" itself using that data

 

What are your thoughts and suggestions?

How do you architect an Async resource manager

 

Thanks

 

** Can a Mod change the title to be spelt correctly please: Asynchronous Resource Manager **

Edited by nickyc95

Share this post


Link to post
Share on other sites
Advertisement
As well as being asynchronous, I wanted to support:
* loading from compressed archives.
* encryption / decryption.
* streaming into memory allocated by the user. E.g. On some platforms, the user may have a block of GPU memory and they want the asset system to stream a texture file directly into the GPU.

So I use multiple callbacks - when the uncompressed file size is know, the first callback lets the user allocate memory for the asset however they like. A second callback lets them know that streaming has completed.
Internally the system may be streaming directly into the users buffer, or streaming compressed blocks into its own internal buffer, and then decompressing into the users buffer.

Share this post


Link to post
Share on other sites

As well as being asynchronous, I wanted to support:
* loading from compressed archives.
* encryption / decryption.
* streaming into memory allocated by the user. E.g. On some platforms, the user may have a block of GPU memory and they want the asset system to stream a texture file directly into the GPU.

So I use multiple callbacks - when the uncompressed file size is know, the first callback lets the user allocate memory for the asset however they like. A second callback lets them know that streaming has completed.
Internally the system may be streaming directly into the users buffer, or streaming compressed blocks into its own internal buffer, and then decompressing into the users buffer.

This is pretty much what I want to support.

I looked through your post on the other thread and have a few questions.

Do you run a single thread for this system or is it part of a larger thread pool?

How do you handle the different file types (archives, etc)?

Would your system work something similar to the following:

//MAIN THREAD
ResourceHandle<T> handle;
pushAsyncRequest(handle, filepath, factory);
return handle;

//THREAD 1
size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(allocSize);
FileStream f = fs.open(filepath);
MemoryStream mem(f); //Loads all the data from the file into memory

//MAIN THREAD - Callback from thread 1
factory.onCompletion(mem); //Deserializes the data to consturct the resource
//Resource is now ready to use & handle is populated 

Could you provide more detail on how you system is laid out (particularly in terms of async loading, i.e. how does your system take the initial sys.m_materialFactory.Load call and output a Material)

If you are using a Material factory to load a material, wont all materials be the same size? What does the measuring stage actually do because wouldn't you just need to allocate a new Material then read the uncompressed data into it? Please let me know if I am missing something  :P

 

Thanks 

Edited by nickyc95

Share this post


Link to post
Share on other sites

Would your system work something similar to the following

Yeah pretty much like that, except in your version there's no point in having the allocate callback as you don't make use of the allocated memory in anyway. You read the file into your own in-memory buffer and then pass your buffer to the factory in the completion callback.
What I do is stream the file directly into the factory's allocation, and then notify the factory once this operation is complete.

//THREAD 1
size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(handle, allocSize);
FileStream f = fs.open(filepath);
f.read(data, allocSize); //Loads all the data from the file into memory

//MAIN THREAD - Callback from thread 1
factory.onCompletion(handle, data); //Deserializes the data to construct the resource
//Resource is now ready to use & handle is populated 

I try to structure most of my formats so that there is minimal deserialization required, and most of them deserialize in-place without extra allocations.
When in-place deserialization isn't possible, then the memory allocated in factory.allocate will be released during factory.onCompletion.

If you are using a Material factory to load a material, wont all materials be the same size? What does the measuring stage actually do because wouldn't you just need to allocate a new Material then read the uncompressed data into it?

Each material may use a different shader, and each shader may declare a different amount of uniform variables and textures. That means that different materials can greatly differ in size :)
Off topic but FWIW, I don't store individual materials as assets, instead I have a "material pack" asset. Each model file will likely contain many materials (for different parts of the model), so when importing a model from an artist, it generates a material-pack asset to go alongside that model's geometry asset.

Do you run a single thread for this system or is it part of a larger thread pool?

Internally on Windows I use the native async file system API (see the lpOverlapped parameter of ReadFileEx)... however, this API still sometimes blocks on occasion (I think when you have too many async operations in flight), so whenever I need to call one of those functions I push a job to a background thread / thread pool so that the main thread(s) won't stall. The main thread(s) poll the blob loader once per frame to check if any file streaming operations have completed, and if so the completion callbacks are triggered (so these happen on the main threads). However, I limit the number of callbacks that can be called per frame, again to avoid stalling the main threads, as certain deserialization operations can be expensive. I plan to develop a system where each factory can provide a hint of how expensive it's callbacks are to help the blob loader manage this.

How do you handle the different file types (archives, etc)?

When I create a blob loader, I pass it an enum value of which kind of implementation I'd like it to create internally (local loose files, packed archive file, remote/network file system). 

how does your system take the initial sys.m_materialFactory.Load call and output a Material

I have something similar to your ResourceHandle<T>, but mine's just called Asset and is basically a union{void* ptr; ptrdiff_t id;};
The different asset types (T) implement wrappers around this common handle that let you gain access to the deserialized type in their own way. e.g. my renderer's actual texture type is TextureId, so the asset version of a texture is called TextureAsset and it has a GetId member function that returns a TextureId (this is equivalent to having ResourceHandle<TextureId>). It's only safe to call GetId once the texture has finished loading though, so there's two mechanisms to determine that.
Instead of allowing you to query Assets (TexureAssets/etc) as to whether they're finished loading or not, I load all assets in batches and associate them with this batch-managing object, called an AssetScope. When loading an object, you need a factory, an asset-scope, an asset-name to load, and a blob-loader. Once you've finished adding load requests into the batch, you can call Close on the AssetScope to tell it that no more assets will be added to this batch.
The first way to tell when the AssetScope is complete is to periodically poll it:

struct Example
{
	AssetScope m_assets;
	TextureAsset* m_foo;
	TextureAsset* m_bar;
	bool m_loaded;
	Example(BlobLoader& loader, TextureFactory& textureFactory)
		: m_loaded()
	{
		AssetName nameFoo("foo.tex");
		AssetName nameBar("foo.tex");
		m_foo = textureFactory.Load( nameFoo, m_assets, loader );
		m_bar = textureFactory.Load( nameBar, m_assets, loader );
		m_assets.Close();
	}
	void Update()
	{
		if( m_assets.Update() == AssetScope::Loaded )
			m_loaded = true;
	}
	bool Loaded() const { return m_loaded; }
	void Draw()
	{
		if( !m_loaded )
			return;
		TextureId foo = m_foo->GetId();
		DrawSomething(foo);
	}
};

And the other is to register a call-back before closing the asset-scope:

struct Example
{
	AssetScope m_assets;
	TextureAsset* m_foo;
	TextureAsset* m_bar;
	bool m_loaded;

	Example(BlobLoader& loader, TextureFactory& textureFactory)
		: m_loaded()
	{
		m_foo = textureFactory.Load( AssetName("foo.tex"), m_assets, loader );
		m_bar = textureFactory.Load( AssetName("foo.tex"), m_assets, loader );
		m_assets.OnLoaded( [this](){ m_loaded = true }, Task::CurrentThread() ); // code to run on completion, and which thread to run it on
		m_assets.Close();
	}

	bool Loaded() const { return m_loaded; }

	void Draw()
	{
		if( !m_loaded )
			return;
		TextureId foo = m_foo->GetId();
		DrawSomething(foo);
	}
};

Share this post


Link to post
Share on other sites

 

Would your system work something similar to the following

Yeah pretty much like that, except in your version there's no point in having the allocate callback as you don't make use of the allocated memory in anyway. You read the file into your own in-memory buffer and then pass your buffer to the factory in the completion callback.
What I do is stream the file directly into the factory's allocation, and then notify the factory once this operation is complete.

//THREAD 1
size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(handle, allocSize);
FileStream f = fs.open(filepath);
f.read(data, allocSize); //Loads all the data from the file into memory

//MAIN THREAD - Callback from thread 1
factory.onCompletion(handle, data); //Deserializes the data to construct the resource
//Resource is now ready to use & handle is populated 

I try to structure most of my formats so that there is minimal deserialization required, and most of them deserialize in-place without extra allocations.
When in-place deserialization isn't possible, then the memory allocated in factory.allocate will be released during factory.onCompletion.

If you are using a Material factory to load a material, wont all materials be the same size? What does the measuring stage actually do because wouldn't you just need to allocate a new Material then read the uncompressed data into it?

Each material may use a different shader, and each shader may declare a different amount of uniform variables and textures. That means that different materials can greatly differ in size :)
Off topic but FWIW, I don't store individual materials as assets, instead I have a "material pack" asset. Each model file will likely contain many materials (for different parts of the model), so when importing a model from an artist, it generates a material-pack asset to go alongside that model's geometry asset.

Do you run a single thread for this system or is it part of a larger thread pool?

Internally on Windows I use the native async file system API (see the lpOverlapped parameter of ReadFileEx)... however, this API still sometimes blocks on occasion (I think when you have too many async operations in flight), so whenever I need to call one of those functions I push a job to a background thread / thread pool so that the main thread(s) won't stall. The main thread(s) poll the blob loader once per frame to check if any file streaming operations have completed, and if so the completion callbacks are triggered (so these happen on the main threads). However, I limit the number of callbacks that can be called per frame, again to avoid stalling the main threads, as certain deserialization operations can be expensive. I plan to develop a system where each factory can provide a hint of how expensive it's callbacks are to help the blob loader manage this.

How do you handle the different file types (archives, etc)?

When I create a blob loader, I pass it an enum value of which kind of implementation I'd like it to create internally (local loose files, packed archive file, remote/network file system). 

how does your system take the initial sys.m_materialFactory.Load call and output a Material

I have something similar to your ResourceHandle<T>, but mine's just called Asset and is basically a union{void* ptr; ptrdiff_t id;};
The different asset types (T) implement wrappers around this common handle that let you gain access to the deserialized type in their own way. e.g. my renderer's actual texture type is TextureId, so the asset version of a texture is called TextureAsset and it has a GetId member function that returns a TextureId (this is equivalent to having ResourceHandle<TextureId>). It's only safe to call GetId once the texture has finished loading though, so there's two mechanisms to determine that.
Instead of allowing you to query Assets (TexureAssets/etc) as to whether they're finished loading or not, I load all assets in batches and associate them with this batch-managing object, called an AssetScope. When loading an object, you need a factory, an asset-scope, an asset-name to load, and a blob-loader. Once you've finished adding load requests into the batch, you can call Close on the AssetScope to tell it that no more assets will be added to this batch.
The first way to tell when the AssetScope is complete is to periodically poll it:

struct Example
{
	AssetScope m_assets;
	TextureAsset* m_foo;
	TextureAsset* m_bar;
	bool m_loaded;
	Example(BlobLoader& loader, TextureFactory& textureFactory)
		: m_loaded()
	{
		AssetName nameFoo("foo.tex");
		AssetName nameBar("foo.tex");
		m_foo = textureFactory.Load( nameFoo, m_assets, loader );
		m_bar = textureFactory.Load( nameBar, m_assets, loader );
		m_assets.Close();
	}
	void Update()
	{
		if( m_assets.Update() == AssetScope::Loaded )
			m_loaded = true;
	}
	bool Loaded() const { return m_loaded; }
	void Draw()
	{
		if( !m_loaded )
			return;
		TextureId foo = m_foo->GetId();
		DrawSomething(foo);
	}
};

And the other is to register a call-back before closing the asset-scope:

struct Example
{
	AssetScope m_assets;
	TextureAsset* m_foo;
	TextureAsset* m_bar;
	bool m_loaded;

	Example(BlobLoader& loader, TextureFactory& textureFactory)
		: m_loaded()
	{
		m_foo = textureFactory.Load( AssetName("foo.tex"), m_assets, loader );
		m_bar = textureFactory.Load( AssetName("foo.tex"), m_assets, loader );
		m_assets.OnLoaded( [this](){ m_loaded = true }, Task::CurrentThread() ); // code to run on completion, and which thread to run it on
		m_assets.Close();
	}

	bool Loaded() const { return m_loaded; }

	void Draw()
	{
		if( !m_loaded )
			return;
		TextureId foo = m_foo->GetId();
		DrawSomething(foo);
	}
};

Fantastic! 

This explains pretty much everything I needed to know, thanks for taking the time to write it  :D

If I have any more questions, I'll post them here  :P

 

Thanks @Hodgman

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!