Sign in to follow this  

Vulkan Techniques for bulk data transfer

Recommended Posts

I'm having as issue with how to efficiently upload large amounts of data to the GPU with Vulkan.  I'm not talking about small constant buffers, or dynamically modified data, but rather the large static assets usually loaded during level loading (textures, models, etc...).

 

In D3D11 it usually consisted of a loop repeatedly calling CreateTexture or CreateBuffer with a corresponding D3D11_SUBRESOURCE_DATA to indicate the data source.  This seemed pretty quick and I imagine the driver knew how to the copy data into device memory (whichever heap/type D3D11_USAGE_IMMUTABLE mapped to) in an optimal way.

 

In Vulkan it seems I have to:

- create a staging VkImage using VK_IMAGE_TILING_LINEAR

- allocate and bind host visible memory to the image

- use map/unmap to upload data

- create final VkImage this time with VK_IMAGE_TILING_OPTIMAL

- allocate and bind device memory to image

- create command buffer

- copy each image subresource from staging to final image

- submit command buffer

- wait on command buffer completion

- repeat 1000x ??

 

I understand the need for a staging buffer when working with dynamic resources that would be modified per frame (or even multiple times per frame).  But when it comes to bulk data transfers it seems the need for a staging buffer as well as the VK_IMAGE_TILING_LINEAR to VK_IMAGE_TILING_OPTIMAL conversion would significantly hinder performance.  As I see it there are a number of options, all of which seem to have issues.

 

1) Use a single large staging image/buffer, use a single command buffer, submit, wait on fence, loop.  This has got to be slow waiting for both the CPU and GPU to sync before uploading the next resource.

 

2) Use multiple staging images/buffers as part of a large memory ring/circular buffer, use multiple command buffers.  This is fairly complex to implement; not a huge deal but for something that seems so simple this feels over engineered.  Also there doesn't seem to be any simple way to predict how much memory to use for staging.  I'd be worried that allocating a large amount of staging memory would affect performance (staging memory could also serve as device memory in some implementations if I understand the spec correctly) and/or lead to out of memory situations.

 

3) Use buffers instead of images for staging both image and buffer data.  Don't know if this helps in any way, but I saw it mentioned in a forum online.  Still have the 'how much memory is too much memory' to allocate issue.

 

I can't find any write-up or presentations on this, any links, slides, talks, etc... that I've come across simply seem to gloss over this whole part.  I was hoping there would be a way I could just dump a large amount of data directly to device memory and then cut it up into its various images/buffers after the fact.  How are you guys approaching this problem?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Forum Statistics

    • Total Topics
      628710
    • Total Posts
      2984323
  • Similar Content

    • By L. Spiro
      Home: https://www.khronos.org/vulkan/
      SDK: http://lunarg.com/vulkan-sdk/
       
      AMD drivers: http://gpuopen.com/gaming-product/vulkan/ (Note that Vulkan support is now part of AMD’s official drivers, so simply getting the latest drivers for your card should give you Vulkan support.)
      NVIDIA drivers: https://developer.nvidia.com/vulkan-driver (Note that Vulkan support is now part of NVIDIA’s official drivers, so simply getting the latest drivers for your card should give you Vulkan support.)
      Intel drivers: http://blogs.intel.com/evangelists/2016/02/16/intel-open-source-graphics-drivers-now-support-vulkan/
       
      Quick reference: https://www.khronos.org/registry/vulkan/specs/1.0/refguide/Vulkan-1.0-web.pdf
      References: https://www.khronos.org/registry/vulkan/specs/1.0/apispec.html
      https://matthewwellings.com/blog/the-new-vulkan-coordinate-system/
       
      GLSL-to-SPIR-V: https://github.com/KhronosGroup/glslang

      Sample code: https://github.com/LunarG/VulkanSamples
      https://github.com/SaschaWillems/Vulkan
      https://github.com/nvpro-samples
      https://github.com/nvpro-samples/gl_vk_chopper
      https://github.com/nvpro-samples/gl_vk_threaded_cadscene
      https://github.com/nvpro-samples/gl_vk_bk3dthreaded
      https://github.com/nvpro-samples/gl_vk_supersampled
      https://github.com/McNopper/Vulkan
      https://github.com/GPUOpen-LibrariesAndSDKs/HelloVulkan
       
      C++: https://github.com/nvpro-pipeline/vkcpp
      https://developer.nvidia.com/open-source-vulkan-c-api

      Getting started: https://vulkan-tutorial.com/
      https://renderdoc.org/vulkan-in-30-minutes.html
      https://www.khronos.org/news/events/vulkan-webinar
      https://developer.nvidia.com/engaging-voyage-vulkan
      https://developer.nvidia.com/vulkan-shader-resource-binding
      https://developer.nvidia.com/vulkan-memory-management
      https://developer.nvidia.com/opengl-vulkan
      https://github.com/vinjn/awesome-vulkan

      Videos: https://www.youtube.com/playlist?list=PLYO7XTAX41FPg08uM_bgPE9HLgDAyzDaZ

      Utilities: https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator (AMD Memory allocator.)
      https://github.com/GPUOpen-LibrariesAndSDKs/Anvil (AMD Miniature Vulkan engine/framework.)

       
      L. Spiro
    • By hiya83
      (Posted this in graphics forum too, which was perhaps the wrong forum for it)
      Hey, I was wondering if on mobile development (Android mainly but iOS as well if you know of it), if there is a GPUView equivalent for whole system debugging so we can figure out if the CPU/GPU are being pipelined efficiently, if there are bubbles, etc. Also slightly tangent question, but do mobile GPU's have a DMA engine exposed as a dedicated Transfer Queue for Vulkan?
      Thanks!
    • By hiya83
      Hey, I was wondering if on mobile development (Android mainly but iOS as well if you know of it), if there is a GPUView equivalent for whole system debugging so we can figure out if the CPU/GPU are being pipelined efficiently, if there are bubbles, etc. Also slightly tangent question, but do mobile GPU's have a DMA engine exposed as a dedicated Transfer Queue for Vulkan?
    • By mark_braga
      I am working on a project which needs to share render targets between Vulkan and DirectX12. I have enabled the external memory extension and now allocate the memory for the render targets by adding the VkExportMemoryInfoKHR to the pNext chain of VkMemoryAllocateInfo. Similarly I have added the VkExternalMemoryImageCreateInfo to the pNext chain of VkImageCreateInfo.
      After calling the get win32 handle function, I get some handle pointer which is not null (I assume it is valid).
      VkExternalMemoryImageCreateInfoKHR externalImageInfo = {}; if (gExternalMemoryExtensionKHR) { externalImageInfo.sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO_KHR; externalImageInfo.pNext = NULL; externalImageInfo.handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_KMT_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_HEAP_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_RESOURCE_BIT_KH imageCreateInfo.pNext = &externalImageInfo; } vkCreateImage(...); VkExportMemoryAllocateInfoKHR exportInfo = { VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO_KHR }; exportInfo.handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_KMT_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_HEAP_BIT_KHR | VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_RESOURCE_BIT_KHR; memoryAllocateInfo.pNext = &exportInfo; vkAllocateMemory(...); VkMemoryGetWin32HandleInfoKHR info = { VK_STRUCTURE_TYPE_MEMORY_GET_WIN32_HANDLE_INFO_KHR, NULL }; info.memory = pTexture->GetMemory(); info.handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D12_RESOURCE_BIT_KHR; VkResult res = vkGetMemoryWin32HandleKHR(vulkanDevice, &info, &pTexture->pSharedHandle); ASSERT(VK_SUCCESS == res); Now when I try to call OpenSharedHandle from a D3D12 device, it crashes inside nvwgf2umx.dll with the integer division by zero error.
      I am now lost and have no idea what the other handle types do.
      For example: How do we get the D3D12 resource from the VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR handle?
      I also found some documentation on this link but it doesn't help much.
      https://javadoc.lwjgl.org/org/lwjgl/vulkan/NVExternalMemoryWin32.html
      This is all assuming the extension works as expected since it has made it to the KHR
    • By dwatt
      I am trying to get vulkan on android working and have run into a big issue. I don't see any validation layers available. I have tried linking their libraries into mine but still no layers. I have tried compiling it into a library of my own but the headers for it are all over the place. Unfortunately , google's examples and tutorials are out of date and don't work for me. Any idea what I have to do to get those layers to work?
  • Popular Now