ComputeShader Performance / Crashes

Started by
2 comments, last by JoeJ 8 years, 2 months ago

Made a (looong) GLSL ComputeShader for Tiled-Deferred rendering. On my laptop with a 2013 nVidia graphics card, it works fine. But now I'm sending the program to some other guys. And as you know, that's always where the headaches start smile.png Can't debug or whatsoever, only guess. I need your experience or guessing-powers to give me some directions!

Guy1 had a nVidia card. Not too old, certainly not new either. Video card driver hanged / crashed when running this particular ComputeShader. Disabling loops "fixed" it:


requires OpenGL 430
#define TILE_SIZE 32
layout (local_size_x = TILE_SIZE, local_size_y = TILE_SIZE) in;
 
shared uint _indxLightsPoint[ MAXLIST_LIGHTS_POINT ];   // Found PointLights, indexes to UBO lightArray
 
...
// 1. Let each pixel inside a tile check ONE pointlight, see if it intersects tile-frustum. Ifso, add to a shared list
uint thrID = gl_LocalInvocationID.x + gl_LocalInvocationID.y * TILE_SIZE;  // Each tasks gets a number (0,1,2, ... 1023)
 
if ( thrID < counts1.x ) { // "count1" comes from a UBO parameter. Would be "2", if there were 2 active lights in the scene
   if ( pointLightIntersects( tileFrustum, lightPoint[ thrID ].posRange ) ) {
      // Add lightIndex to list
      uint index = atomicAdd( _cntLightsPoint, 1 );
      if ( index < MAXLIST_LIGHTS_POINT ) 
           _indxLightsPoint[ index ] = thrID;
   }
}
 
...
barrier();
...
 
// 2. Loop through the lights we found
 
for (uint i=0; i < _cntLightsPoint; i++) {
    uint index = _indxLightsPoint[i];
    addPointLight( brdf, surf, lightPoint[index] );
} // for

Compiles, starts, hangs the video-driver. If I simplify all this code to a fixed " addPointLight( ... lightPoint[ 0 ] )", it works. And a damn lot faster as well (even though I only had 1 or 2 lights in the scene anyway). If I re-enable "barrier" or some of the atomic operations, the FPS crumbles again. My first thought was that the "FOR LOOP" went crazy, counting to an extreme high number. But even if I put a hard-coded number here, it still crashes. The other suspect might be an out-of-range array read, but I can't see how.

Could it be that "older" cards (2010..2012) have issues with (GLSL) Barriers or Atomic operations? Or maybe the hard-coded Tilesize (32x32) is too big? Although I would expect a compiler crash in that case.

Guy1 now has a new AMD card. But it seems it doesn't support some OpenGL 4.5.0 features (though all shaders use 430). Got stranded after that.

Guy2 had a 2011 nVidia card, don't know what exactly. Everything works, but graphics seem more blurry (anisotropic / mipmapping settings?). Moroever, framerate is horrible. Mines is ~50 .. 60 FPS at a larger resolution, his is 5. I expected a drop, but not that much. As usual there could be a billio things wrong, but my main suspects are:

- ComputeShader setup (tilesize 32x32 too big)

- ComputeShader operations (atomicAdd / atomicMin / atomicMax / FOR LOOP / Barrier )

- I assume 24+ texture units are available ( ie "layout(binding=20) uniform sampler2D gBufferXYZ;" ). I know older cards only have 16 or so. But again I would expect a crash then.

- Not using glMemoryBarrier( GL_ALL_BARRIER_BITS ); (properly), prior or after calling the CS

My guts say to replace the ComputeShader with good old Fragment shaders and such. Then again it just works well on my own computer. And since its quite a job to change, it would suck if something very different turns out to be the party-crasher.

Ciao!

Advertisement
You modify shared memory (_indxLightsPoint[ index ] = thrID),
you do a barrier, but you forget to do a memory marrier on shared memory as well.
You read shared memory (index = _indxLightsPoint), but it is not guaranteed that all threads see the expected thrID.

Maybe that's it. I'd not give up so soon because you have no shared memory in fragment shaders.
Personally i gave up on OpenGL compute shader because OpenCL was two times faster on Nvidia ans slightly faster on AMD 1-2 years ago.

Edit:
For me it was absolutely necessery to stop the compiler from unrolling loops (forgot the command)
The compiler did not bother to unroll loops with > 1000 iterations smile.png

- ComputeShader setup (tilesize 32x32 too big)


It's always worth to try out, different hardware, differnt results. I'd assume 8*8 or 16*16 is better than 32*32.
On OpenCL the maximum for ATI is 512, but OpenGL spec requires a minimum of 1024, so i guess it's a slowdown for ATI to sync 1024 threads.
The hardware minimum for ATI is 64, NV 32. So in practice choose 64, 128 and 256 depending mostly on register usage.

Thanks for taking time to wrestle through my code pieces Joe!

>> you forget to do a memory barrier on shared memory as well.

All right. Adding "memoryBarrierShared()" in addition to "barrier()" would do the job (to ensure the index-array is done filling before starting the second half)?

Btw, besides crashes, is it possible that bad/lacking usage of the barrier as suggested can cause such a huge slowdown? Like I said, on my computer all seems fine, another one works as expected as well, but just very slow.

>> because OpenCL was two times faster on Nvidia ans slightly faster on AMD 1-2 years ago

Now that concerns me. Especially because I used OpenCL before, removed it completely from the engine, and swapped it for OpenCL (easier integration, more consistency)... Doh!

Is it safe to assume that modern/future cards will overcome these performance issues? Otherwise I can turn my Deferred Rendering approach back to an "old" additive style. Anyone experience if Tiled Difference Rendering is that much of a win? And then I'm talking about indoor scenes which have relative much lights, but certainly not hundreds or thousands.

The crappy part is that I'm adapting code to support older cards now, even though I'm far away from a release, so maybe I shouldn't put too much energy on that and bet on future hardware.

>> Unroll

I suppose that can't happen if the size isn't hardcoded (counts.x comes from an outside (CPU) variable)?

Well, let's try the shared-barrier, different workgroup size, and avoiding unrolling. And see if these video-cards start smiling... But I'm afraid not hehe.

>> you forget to do a memory barrier on shared memory as well.
All right. Adding "memoryBarrierShared()" in addition to "barrier()" would do the job (to ensure the index-array is done filling before starting the second half)?
Btw, besides crashes, is it possible that bad/lacking usage of the barrier as suggested can cause such a huge slowdown? Like I said, on my computer all seems fine, another one works as expected as well, but just very slow.


Yes, barrier syncs only the codeflow, so you need the memory barriers to ensure all writes are done as well.
This could cause e.g. a random huge number of lights and cause slow down / locking driver (but this seems not possible for your code).

On the CPU side, when you need te be sure a shader is done, the only thing worked for me was using a Fence.
glMemoryBarrier() or similar alone was not enough. I've had the feeling this was an Nvidia driver bug.

On AMD there was the problem that i had to remove all deprecated gl functions (like glVertex).
Otherwise compute shader produced wrong results. Checking GL errors helped to find those functions.



Is it safe to assume that modern/future cards will overcome these performance issues? Otherwise I can turn my Deferred Rendering approach back to an "old" additive style. Anyone experience if Tiled Difference Rendering is that much of a win?


I don't know if NV has improved their drivers. But they do support CL 1.2 now, with better OpenGL sharing.
I'd give it a try again to compare performance and troubles.
And looking at what 2.0 can do: GPU controls itself without those costly CPU <-> transfers just to get a number and start another kernel... that's exactly what we need.
(I don't know how DX12 or Vulkan can / will compete here)

Compute is worth it if you have an algorithm that can be made to profit from shared memory.
If you read the same stuff from global memory more than once, you can cache that data in shared memory.
You can build acceleration structures in shared memory to avoid typical fragment shader brute force crap. etc...
Just bang your head against the wall long enough until you get an idea how to make use of it smile.png

For the Unroll you're right. (Even for small loops disabling uroll is a win sometimes)

Extremely helpfull is a GPU profiler - ("Nsight"?)

This topic is closed to new replies.

Advertisement