# Asynchronus Resource Manager

This topic is 580 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hi,

So I am building a resource manager for my engine and I have currently gotten stumped as to what approach to take with regards async behaviour.

I have an idea at the moment but i'm not sure if it would be a good approach. (I will just post the ideas for async as it will be very similar for non-async)

Approach

• Request to the resource manager to load a file asynchronously
• The resource manager creates a new AsyncFileRequest with a callback to load the data from the file, this request is placed in a queue that is handled by the file manager. When the manager picks up the request, it will open the disk file (possibly loaded into a memory stream asynchronously)
• Then calls the callback so that the data can be extracted from the file and the resource can "load" itself using that data

What are your thoughts and suggestions?

How do you architect an Async resource manager

Thanks

** Can a Mod change the title to be spelt correctly please: Asynchronous Resource Manager **

Edited by nickyc95

##### Share on other sites

in the neighbouring forum

Thats interesting, thanks! Didn't realise it was a similar topic (because of the title).

The approach that I have listed is similar to what Hodgman posted (#12).

##### Share on other sites
As well as being asynchronous, I wanted to support:
* encryption / decryption.
* streaming into memory allocated by the user. E.g. On some platforms, the user may have a block of GPU memory and they want the asset system to stream a texture file directly into the GPU.

So I use multiple callbacks - when the uncompressed file size is know, the first callback lets the user allocate memory for the asset however they like. A second callback lets them know that streaming has completed.
Internally the system may be streaming directly into the users buffer, or streaming compressed blocks into its own internal buffer, and then decompressing into the users buffer.

##### Share on other sites

As well as being asynchronous, I wanted to support:
* encryption / decryption.
* streaming into memory allocated by the user. E.g. On some platforms, the user may have a block of GPU memory and they want the asset system to stream a texture file directly into the GPU.

So I use multiple callbacks - when the uncompressed file size is know, the first callback lets the user allocate memory for the asset however they like. A second callback lets them know that streaming has completed.
Internally the system may be streaming directly into the users buffer, or streaming compressed blocks into its own internal buffer, and then decompressing into the users buffer.

This is pretty much what I want to support.

I looked through your post on the other thread and have a few questions.

Do you run a single thread for this system or is it part of a larger thread pool?

How do you handle the different file types (archives, etc)?

Would your system work something similar to the following:

//MAIN THREAD
ResourceHandle<T> handle;
pushAsyncRequest(handle, filepath, factory);
return handle;

size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(allocSize);
FileStream f = fs.open(filepath);
MemoryStream mem(f); //Loads all the data from the file into memory

factory.onCompletion(mem); //Deserializes the data to consturct the resource
//Resource is now ready to use & handle is populated



Could you provide more detail on how you system is laid out (particularly in terms of async loading, i.e. how does your system take the initial sys.m_materialFactory.Load call and output a Material)

If you are using a Material factory to load a material, wont all materials be the same size? What does the measuring stage actually do because wouldn't you just need to allocate a new Material then read the uncompressed data into it? Please let me know if I am missing something  :P

Thanks

Edited by nickyc95

##### Share on other sites

Would your system work something similar to the following

Yeah pretty much like that, except in your version there's no point in having the allocate callback as you don't make use of the allocated memory in anyway. You read the file into your own in-memory buffer and then pass your buffer to the factory in the completion callback.
What I do is stream the file directly into the factory's allocation, and then notify the factory once this operation is complete.

//THREAD 1
size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(handle, allocSize);
FileStream f = fs.open(filepath);

factory.onCompletion(handle, data); //Deserializes the data to construct the resource
//Resource is now ready to use & handle is populated 

I try to structure most of my formats so that there is minimal deserialization required, and most of them deserialize in-place without extra allocations.
When in-place deserialization isn't possible, then the memory allocated in factory.allocate will be released during factory.onCompletion.

If you are using a Material factory to load a material, wont all materials be the same size? What does the measuring stage actually do because wouldn't you just need to allocate a new Material then read the uncompressed data into it?

Each material may use a different shader, and each shader may declare a different amount of uniform variables and textures. That means that different materials can greatly differ in size :)
Off topic but FWIW, I don't store individual materials as assets, instead I have a "material pack" asset. Each model file will likely contain many materials (for different parts of the model), so when importing a model from an artist, it generates a material-pack asset to go alongside that model's geometry asset.

Do you run a single thread for this system or is it part of a larger thread pool?

Internally on Windows I use the native async file system API (see the lpOverlapped parameter of ReadFileEx)... however, this API still sometimes blocks on occasion (I think when you have too many async operations in flight), so whenever I need to call one of those functions I push a job to a background thread / thread pool so that the main thread(s) won't stall. The main thread(s) poll the blob loader once per frame to check if any file streaming operations have completed, and if so the completion callbacks are triggered (so these happen on the main threads). However, I limit the number of callbacks that can be called per frame, again to avoid stalling the main threads, as certain deserialization operations can be expensive. I plan to develop a system where each factory can provide a hint of how expensive it's callbacks are to help the blob loader manage this.

How do you handle the different file types (archives, etc)?

When I create a blob loader, I pass it an enum value of which kind of implementation I'd like it to create internally (local loose files, packed archive file, remote/network file system).

how does your system take the initial sys.m_materialFactory.Load call and output a Material

I have something similar to your ResourceHandle<T>, but mine's just called Asset and is basically a union{void* ptr; ptrdiff_t id;};
The different asset types (T) implement wrappers around this common handle that let you gain access to the deserialized type in their own way. e.g. my renderer's actual texture type is TextureId, so the asset version of a texture is called TextureAsset and it has a GetId member function that returns a TextureId (this is equivalent to having ResourceHandle<TextureId>). It's only safe to call GetId once the texture has finished loading though, so there's two mechanisms to determine that.
The first way to tell when the AssetScope is complete is to periodically poll it:

struct Example
{
AssetScope m_assets;
TextureAsset* m_foo;
TextureAsset* m_bar;
{
AssetName nameFoo("foo.tex");
AssetName nameBar("foo.tex");
m_assets.Close();
}
void Update()
{
}
void Draw()
{
return;
TextureId foo = m_foo->GetId();
DrawSomething(foo);
}
};

And the other is to register a call-back before closing the asset-scope:

struct Example
{
AssetScope m_assets;
TextureAsset* m_foo;
TextureAsset* m_bar;

{
m_assets.Close();
}

void Draw()
{
return;
TextureId foo = m_foo->GetId();
DrawSomething(foo);
}
};

##### Share on other sites

Would your system work something similar to the following

Yeah pretty much like that, except in your version there's no point in having the allocate callback as you don't make use of the allocated memory in anyway. You read the file into your own in-memory buffer and then pass your buffer to the factory in the completion callback.
What I do is stream the file directly into the factory's allocation, and then notify the factory once this operation is complete.

//THREAD 1
size_t allocSize = calcAllocSize(filepath);
void* data = factory.allocate(handle, allocSize);
FileStream f = fs.open(filepath);

factory.onCompletion(handle, data); //Deserializes the data to construct the resource
//Resource is now ready to use & handle is populated 

I try to structure most of my formats so that there is minimal deserialization required, and most of them deserialize in-place without extra allocations.
When in-place deserialization isn't possible, then the memory allocated in factory.allocate will be released during factory.onCompletion.

If you are using a Material factory to load a material, wont all materials be the same size? What does the measuring stage actually do because wouldn't you just need to allocate a new Material then read the uncompressed data into it?

Each material may use a different shader, and each shader may declare a different amount of uniform variables and textures. That means that different materials can greatly differ in size :)
Off topic but FWIW, I don't store individual materials as assets, instead I have a "material pack" asset. Each model file will likely contain many materials (for different parts of the model), so when importing a model from an artist, it generates a material-pack asset to go alongside that model's geometry asset.

Do you run a single thread for this system or is it part of a larger thread pool?

Internally on Windows I use the native async file system API (see the lpOverlapped parameter of ReadFileEx)... however, this API still sometimes blocks on occasion (I think when you have too many async operations in flight), so whenever I need to call one of those functions I push a job to a background thread / thread pool so that the main thread(s) won't stall. The main thread(s) poll the blob loader once per frame to check if any file streaming operations have completed, and if so the completion callbacks are triggered (so these happen on the main threads). However, I limit the number of callbacks that can be called per frame, again to avoid stalling the main threads, as certain deserialization operations can be expensive. I plan to develop a system where each factory can provide a hint of how expensive it's callbacks are to help the blob loader manage this.

How do you handle the different file types (archives, etc)?

When I create a blob loader, I pass it an enum value of which kind of implementation I'd like it to create internally (local loose files, packed archive file, remote/network file system).

how does your system take the initial sys.m_materialFactory.Load call and output a Material

I have something similar to your ResourceHandle<T>, but mine's just called Asset and is basically a union{void* ptr; ptrdiff_t id;};
The different asset types (T) implement wrappers around this common handle that let you gain access to the deserialized type in their own way. e.g. my renderer's actual texture type is TextureId, so the asset version of a texture is called TextureAsset and it has a GetId member function that returns a TextureId (this is equivalent to having ResourceHandle<TextureId>). It's only safe to call GetId once the texture has finished loading though, so there's two mechanisms to determine that.
The first way to tell when the AssetScope is complete is to periodically poll it:

struct Example
{
AssetScope m_assets;
TextureAsset* m_foo;
TextureAsset* m_bar;
{
AssetName nameFoo("foo.tex");
AssetName nameBar("foo.tex");
m_assets.Close();
}
void Update()
{
}
void Draw()
{
return;
TextureId foo = m_foo->GetId();
DrawSomething(foo);
}
};

And the other is to register a call-back before closing the asset-scope:

struct Example
{
AssetScope m_assets;
TextureAsset* m_foo;
TextureAsset* m_bar;

{
m_assets.Close();
}

void Draw()
{
return;
TextureId foo = m_foo->GetId();
DrawSomething(foo);
}
};

Fantastic!

This explains pretty much everything I needed to know, thanks for taking the time to write it  :D

If I have any more questions, I'll post them here  :P

Thanks @Hodgman

1. 1
2. 2
Rutin
19
3. 3
khawk
18
4. 4
5. 5
A4L
11

• 9
• 12
• 16
• 26
• 10
• ### Forum Statistics

• Total Topics
633769
• Total Posts
3013754
×