# DX11 [DX11] Handling multiple GPUs

This topic is 2824 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I did alot of googling on the subject of multiple GPUs. Both NVidia and ATI have presentations available, but all they give are guidelines on how to structure your rendering code to make AFR (alternate frame rendering) work efficiently.

My goal is quite different. I would like to perform standard rendering on GPU1, and at the same time, have compute shader code running on GPU2 (ie no AFR).

Does anyone have any pointers / links / advice on how I can test a setup like this? My current hardware configuration consists of 2 crossfired HD5750's.

Is this even possible through the DX11 API? If I were able to achieve this, would I even be able to share compute shader results from GPU2 to GPU1 without having to read results back to the CPU first?

Thanks,
Chris

##### Share on other sites
Can't you just pass separate adapter identifiers for two D3D 11 devices? That would be the most straightforward way, but it will require you to un-crossfire your cards.

##### Share on other sites
Hmm... true, that would do that job, though I was hoping for some way to share data between the GPUs through the crossfire / SLI bridge. That is unless there is a very fast way of getting the data to the other GPU without it?

##### Share on other sites
Not really, that's why there's a bridge. But Crossfire and SLI merge the cards into a single logical GPU, so you can't allocate tasks to one or the other.

##### Share on other sites
Would you mind sharing the results of this experimentation? It sounds quite interesting to me.

##### Share on other sites
Sure thing.

What I plan on doing is testing two setups:

1) Crossfire the cards as normal. Do my compute shader stuff, followed by my rendering stuff in series (paying attention to guidelines given by ATI for alternate frame rendering)

2) Uncrossfire the cards. Create two separate D3D devices, one per card. Do all my compute stuff on one device, and all my rendering stuff on the other device. As soon as results become available from the compute device, upload to CPU, then load onto rendering GPU for use.

If you're curious, the "compute stuff" is a patch based radiosity solver. So while it doesn't NEED to be completely in sync with the rendering, it would certainly be more visually pleasing to see lighting updates as the light moves around, rather than having some delay in getting results.

I expect that 1) will give the most visually pleasing results, but the rendering might be bogged down if the solving step takes longer than theres room for. 2) would allow me to have the radiosity solved in a "separate thread" per se, independent of the rendering, which should make moving the camera around more snappy, but with delayed lighting results (which could turn out to be ok)

##### Share on other sites
:) this is exactly the project I'm doing. If you uncrossfire the cards, you can iterate through the adapter list, create one device on each card, then copy buffers between the two using UpdateSubresource(). You have to copy the UAV buffer to a readable buffer, and then read from that, and update the render GPU with the info that you get. If they are crossfired together, DirectX sees them as one adapter, and you don't have very much control over distribution between the two.

I managed to get the uncrossfire version working with a particle simulation, where you compute on one GPU and render on another GPU. on ATI, the performance boost wasn't very high, but with two NVIDIAs, you get a huge boost (probably because of the difference in memory speed).

I have the code snippet for copying the buffers between two devices posted here:

[D3D11] Map() from two different devices causing system hangup / freeze

if you run into a similar problem as the one I posted, let me know, I've been pulling my hair out for weeks trying to get two compute devices to work concurrently.

• 10
• 17
• 9
• 14
• 41
• ### Similar Content

• So I've been playing around today with some things in D3D 11.1, specifically the constant buffer offset stuff.  And just FYI, I'm doing this in C# with SharpDX (latest version).
I got everything set up, I have my constant buffer populating with data during each frame, and calling VSSetConstantBuffers1 and passing in the offset/count as needed.
But, unfortunately, I get nothing on my screen.  If I go back to using the older D3D11 SetConstantBuffers method (without the offset/count), everything works great.
I get nothing from the D3D runtime debug spew, and a look in the graphics debugger stuff tells me that my constant buffer does indeed have data at the offsets that I'm providing.  And the data (World * Projection matrix) is correct at each offset.  The offsets, according again to the graphics debugger, are correct.
I could be using it incorrectly, but what little (and seriously, there's not a lot) info I found seems to indicate that I'm doing it correctly.  But here's my workflow (I'd post code, but it's rather massive):
Frame #0:
Map constant buffer with discard Write matrix at offset 0, count 64 Unmap VSSetConstantBuffers1(0, 1, buffers, new int[] { offset }, new int[] { count });  // Where offset is the offset above, same with count Draw single triangle Frame #1:
Map constant buffer with no-overwrite Write matrix at offset 64, count 64. Unmap VSSetConstantBuffers1(0, 1, buffers, new int[] { offset }, new int[] { count });  // Where offset is the offset above, same with count Draw single triangle Etc... it repeats until the end of the buffer, and starts over with a discard when the buffer is full.
Has anyone ever used these offset cbuffer functions before?  Can you help a brother out?
Edit:
I've added screenshots of what I'm seeing the VS 2017 graphics debugger.  As I said before, if I use the old VSSetConstantBuffers method, it works like a charm and I see my triangle.

• By cozzie
Hi all,
It's been a while since I've been working on my HLSL shaders, and found out I'm not 100% sure if I'm applying gamma correctness correctly. So here's what I do:
- create backbuffer in this format: DXGI_FORMAT_R8G8B8A8_UNORM_SRGB
- source textures (DDS) are always in SRGB format
- this way the textures should be gamma correct, because DX11 helps me out here
Now my question is about material and light colors. I'm not sure if I need to convert those to linear space. The colors are handpicked on screen, so I guess gamma correct. Below are 2 screenshots, the darker is including converting those colors (return float4(linearColor.rgb * linearColor.rgb, linearColor.a);), in the lighter shot I didn't do this conversion.
These are the properties of the brick material and the light source (there are no other lightsources in the scene, also no global ambient):
Material:
CR_VECTOR4(0.51f, 0.26f, 0.22f, 1.0f), // ambient CR_VECTOR4(0.51f, 0.26f, 0.22f, 1.0f), // diffuse RGB + alpha CR_VECTOR4(0.51f, 0.26f, 0.22f, 4.0f)); // specular RGB + power Directional light:
mDirLights[0].Ambient = CR_VECTOR4(0.1f, 0.1f, 0.1f, 1.0f); mDirLights[0].Diffuse = CR_VECTOR4(0.75f, 0.75f, 0.75f, 1.0f); mDirLights[0].Specular = CR_VECTOR4(1.0f, 1.0f, 1.0f, 16.0f); mDirLights[0].Direction = CR_VECTOR3(0.0f, 1.0f, 0.0f);
So in short, should I or should I not do this conversion in the lighting calculation in the shader? (and/or what else are you seeing :))
Note that I don't do anything with the texture color, after it's fetched in the shader (no conversions), which I believe is correct.

• By chiffre
Introduction:
In general my questions pertain to the differences between floating- and fixed-point data. Additionally I would like to understand when it can be advantageous to prefer fixed-point representation over floating-point representation in the context of vertex data and how the hardware deals with the different data-types. I believe I should be able to reduce the amount of data (bytes) necessary per vertex by choosing the most opportune representations for my vertex attributes. Thanks ahead of time if you, the reader, are considering the effort of reading this and helping me.
I found an old topic that shows this is possible in principal, but I am not sure I understand what the pitfalls are when using fixed-point representation and whether there are any hardware-based performance advantages/disadvantages.
(TLDR at bottom)
The Actual Post:
To my understanding HLSL/D3D11 offers not just the traditional floating point model in half-,single-, and double-precision, but also the fixed-point model in form of signed/unsigned normalized integers in 8-,10-,16-,24-, and 32-bit variants. Both models offer a finite sequence of "grid-points". The obvious difference between the two models is that the fixed-point model offers a constant spacing between values in the normalized range of [0,1] or [-1,1], while the floating point model allows for smaller "deltas" as you get closer to 0, and larger "deltas" the further you are away from 0.
To add some context, let me define a struct as an example:
struct VertexData { float[3] position; //3x32-bits float[2] texCoord; //2x32-bits float[3] normals; //3x32-bits } //Total of 32 bytes Every vertex gets a position, a coordinate on my texture, and a normal to do some light calculations. In this case we have 8x32=256bits per vertex. Since the texture coordinates lie in the interval [0,1] and the normal vector components are in the interval [-1,1] it would seem useful to use normalized representation as suggested in the topic linked at the top of the post. The texture coordinates might as well be represented in a fixed-point model, because it seems most useful to be able to sample the texture in a uniform manner, as the pixels don't get any "denser" as we get closer to 0. In other words the "delta" does not need to become any smaller as the texture coordinates approach (0,0). A similar argument can be made for the normal-vector, as a normal vector should be normalized anyway, and we want as many points as possible on the sphere around (0,0,0) with a radius of 1, and we don't care about precision around the origin. Even if we have large textures such as 4k by 4k (or the maximum allowed by D3D11, 16k by 16k) we only need as many grid-points on one axis, as there are pixels on one axis. An unsigned normalized 14 bit integer would be ideal, but because it is both unsupported and impractical, we will stick to an unsigned normalized 16 bit integer. The same type should take care of the normal vector coordinates, and might even be a bit overkill.
struct VertexData { float[3] position; //3x32-bits uint16_t[2] texCoord; //2x16bits uint16_t[3] normals; //3x16bits } //Total of 22 bytes Seems like a good start, and we might even be able to take it further, but before we pursue that path, here is my first question: can the GPU even work with the data in this format, or is all I have accomplished minimizing CPU-side RAM usage? Does the GPU have to convert the texture coordinates back to a floating-point model when I hand them over to the sampler in my pixel shader? I have looked up the data types for HLSL and I am not sure I even comprehend how to declare the vertex input type in HLSL. Would the following work?
struct VertexInputType { float3 pos; //this one is obvious unorm half2 tex; //half corresponds to a 16-bit float, so I assume this is wrong, but this the only 16-bit type I found on the linked MSDN site snorm half3 normal; //same as above } I assume this is possible somehow, as I have found input element formats such as: DXGI_FORMAT_R16G16B16A16_SNORM and DXGI_FORMAT_R16G16B16A16_UNORM (also available with a different number of components, as well as different component lengths). I might have to avoid 3-component vectors because there is no 3-component 16-bit input element format, but that is the least of my worries. The next question would be: what happens with my normals if I try to do lighting calculations with them in such a normalized-fixed-point format? Is there no issue as long as I take care not to mix floating- and fixed-point data? Or would that work as well? In general this gives rise to the question: how does the GPU handle fixed-point arithmetic? Is it the same as integer-arithmetic, and/or is it faster/slower than floating-point arithmetic?
Assuming that we still have a valid and useful VertexData format, how far could I take this while remaining on the sensible side of what could be called optimization? Theoretically I could use the an input element format such as DXGI_FORMAT_R10G10B10A2_UNORM to pack my normal coordinates into a 10-bit fixed-point format, and my verticies (in object space) might even be representable in a 16-bit unsigned normalized fixed-point format. That way I could end up with something like the following struct:
struct VertexData { uint16_t[3] pos; //3x16bits uint16_t[2] texCoord; //2x16bits uint32_t packedNormals; //10+10+10+2bits } //Total of 14 bytes Could I use a vertex structure like this without too much performance-loss on the GPU-side? If the GPU has to execute some sort of unpacking algorithm in the background I might as well let it be. In the end I have a functioning deferred renderer, but I would like to reduce the memory footprint of the huge amount of vertecies involved in rendering my landscape.
TLDR: I have a lot of vertices that I need to render and I want to reduce the RAM-usage without introducing crazy compression/decompression algorithms to the CPU or GPU. I am hoping to find a solution by involving fixed-point data-types, but I am not exactly sure how how that would work.
• By cozzie
Hi all,
I was wondering it it matters in which order you draw 2D and 3D items, looking at the BeginDraw/EndDraw calls on a D2D rendertarget.
The order in which you do the actual draw calls is clear, 3D first then 2D, means the 2D (DrawText in this case) is in front of the 3D scene.
The question is mainly about when to call the BeginDraw and EndDraw.
Note that I'm drawing D2D stuff through a DXGI surface linked to the 3D RT.
Option 1:
A - Begin frame, clear D3D RT
B - Draw 3D
C - BeginDraw D2D RT
D - Draw 2D
E - EndDraw D2D RT
F - Present
Option 2:
A - Begin frame, clear D3D RT + BeginDraw D2D RT
B - Draw 3D
C - Draw 2D
D - EndDraw D2D RT
E- Present
Would there be a difference (performance/issue?) in using option 2? (versus 1)
Any input is appreciated.

• Do you know any papers that cover custom data structures like lists or binary trees implemented in hlsl without CUDA that work perfectly fine no matter how many threads try to use them at any given time?
• By cozzie
Hi all,
Last week I noticed that when I run my test application(s) in Renderdoc, it crashes when it enable my code that uses D2D/DirectWrite. In Visual Studio no issues occur (debug or release), but when I run the same executable in Renderdoc, it crashes somehow (assert of D2D rendertarget or without any information). Before I spend hours on debugging/ figuring it out, does someone have experience with this symptom and/or know if Renderdoc has known issues with D2D? (if so, that would be bad news for debugging my application in the future );
I can also post some more information on what happens, code and which code commented out, eliminates the problems (when running in RenderDoc).
Any input is appreciated.

• Hi Guys,
I understand how to create input layouts etc... But I am wondering is it at all possible to derive an input layout from a shader and create the input layout directly from this? (Rather than manually specifying the input layout format?)

• Hello,
I am trying to recreate a feature that exists in Unity which is called Stretched Billboarding. However I am having a hard time figuring out how to use a particle velocity to rotate and stretch the particle-quad accordingly.
Here's a screenie of unity's example:

Depending on the velocity of the particle, the quad rotates and stretches, but it is still always facing the camera.
In my current solution I have normal billboarding and velocities and particle-local rotations are working fine.
I generate my quads in a geometry-shader, if that makes any difference.
So does anyone have any thoughts of how to achieve this?
Best regards
Hampus

• Hi guys,
I am having troubles implementing a directional light in HLSL.
I have been following this guide https://www.gamasutra.com/view/feature/131275/implementing_lighting_models_with_.php?page=2 but the quad is rendering pure black no matter where the light vector is pointing.
My quad is in the format of position, texcoord, & normal.
This is the shader so far,
cbuffer ModelViewProjectionConstantBuffer : register(b0) { float4x4 worldmat; float4x4 worldviewmat; float4x4 worldviewprojmat; float4 vecLightDir; } struct VS_OUTPUT { float4 Pos : SV_POSITION; float3 Light : TEXCOORD0; float3 Norm : TEXCOORD1; }; VS_OUTPUT vs_main(float4 position : POSITION, float2 texcoord : TEXCOORD, float3 normal : NORMAL) { float4 newpos; newpos = mul(position, worldmat); newpos = mul(newpos, worldviewmat); newpos = mul(newpos, worldviewprojmat); // return newpos; VS_OUTPUT Out = (VS_OUTPUT)0; Out.Pos = newpos;// mul(newpos, matWorldViewProj); // transform Position Out.Light = vecLightDir; // output light vector Out.Norm = normalize(mul(normal, worldmat)); // transform Normal and normalize it return Out; } float4 ps_main(float3 Light: TEXCOORD0, float3 Norm : TEXCOORD1) : SV_TARGET { float4 diffuse = { 1.0f, 0.0f, 0.0f, 1.0f }; float4 ambient = { 0.1, 0.0, 0.0, 1.0 }; //return float4(1.0f, 1.0f, 1.0f, 1.0f); return ambient + diffuse * saturate(dot(Light, Norm)); } Any help would be truly appreciated as this is the only real area of DX11 that I really have difficulties in.
• By isu diss
How do I fill the gap between sky and terrain? Scaling the terrain or procedural terrain rendering?

• ### Forum Statistics

• Total Topics
631073
• Total Posts
2997754
×