So I've read about the modern gpu architecture and also about the compute shader threading model but what I don't fully understand yet is their relationship to each other.

From what I understand the gpu has several SIMD lanes (or pipes) each featuring several SIMD processing units (shader cores / threads /...), which also have some fetch/decode, ALUs, and execution contexts in them.

I've seen this simple example for a compute kernel where they multiply the values of an array of 2000 elements by 2 and store it back again.

The way they used the threading model was launching the dispatch with (20, 1, 1 ) and using numthreads[100, 1, 1]. Which would be 20 threadgroups, each having 100 threads that execute the compute kernel.

Now what I don't quite understand is what the relationship of the number of threadgroups (wavefronts/warps?) and theads (SIMD units?) are.

If these are a 1:1 mapping, does that mean that I can manually tell the gpu how many threads a wavefront should have ? Sounds unlikely...

If that's not the case, then under what criteria do I calculate these numbers for what I'm trying to accomplish with the compute shader ?

Using the example above, why not use 200 threadgroups with each 10 threads ? Or how about something more random like 5 threadgroups and 400 threads each ? How do I know what the most efficient number of threadgroups / threads are for a certain task ? And how does this map to the actual hardware ?

does that mean that I can manually tell the gpu how many threads a wavefront should have ? Sounds unlikely...

You can -- threadblock * threads per block = # threads per wave. Up to some maximum #of blocks * max #of threads per block.

what criteria do I calculate these numbers for what I'm trying to accomplish with the compute shader

You will probably have to try different settings and profile for the range of inputs, though you can make some guesses/decisions ahead of time based on memory needs, register usage, etc..

why not use 200 threadgroups with each 10 threads ? Or how about something more random like 5 threadgroups and 400 threads each ? How do I know what the most efficient number of threadgroups / threads are for a certain task ?

You want to fully utilize the hardware (fast memory, registers and instruction pipeline) to minimize time needed to do the work. Depends on how many cores you have on the GPU, how much cache, available registers, ...and other stuff. You can only run so many threads in parallel, so you want those threads to be utilizing the resources allocated to them otherwise you are leaving hardware idle (ie, use fewer threads that do more work) . But if the threads require too many resources, then fewer threads can actually run in parallel and speed takes a big hit (so use more threads that do less). You have to find the right balance, probably through testing. Also, whether or not threads need to pass data to each other will factor strongly in the decision, since inter-block communication is problematic.

And how does this map to the actual hardware ?

A few years ago, I found some great diagrams that showed this...keep searching, especially sites of companies that produce GPUs.

So they are actually more like virtual threads and the hardware is using its own partitioning of wavefronts/warps,... to do the work...?

And in the case of AMD hardware a good number of threads to use then would be a multiple of 64 ? What about the group size ? Is that just a case of trying to make it fit ?

This means that at any given time, on AMD hardware, 64 threads are executing the SAME instruction at the same time in lock step.

Correct, but I believe that's per compute unit, not per-gpu.

A given GPU typically has several compute units. A 7970 has 32 of them. Inside the compute unit, currently, those 64 thread are organized as 4x 16-wide SIMD execution units. NV's SMX units are similar in concept, but arranged differently. This is also the level that many resources are shared (memory controller access, texture units, ROPs), so while leveraging this information to run code better may risk over-tuning things, its important to understand for some performance characteristics.

Each CU in a 7970 actually executes 4 wavefronts concurrently. Each wavefront is mapped to a 16-wide SIMD unit, and instructions are executed 16 threads at time. Not that this changes much, but who doesn't like being pedantic.

Anyway these would be my TL:DR guidlines for thread groups:

1. Always prefer thread group sizes that are a multiple of the warp/wavefront count of your hardware. If you're targetting both AMD and Nvidia, a multiple of 64 is safe for both.

2. You usually want more than one wavefront/warp per thread group. Having more lets the hardware swap out warps/wavefronts to hide latency. 128-256 is usually a good place to aim.

3. Don't use too much shared memory, it kills occupancy