We have what we call a "safe sleep" class that picks either Sleep(0) or Sleep(1) based on an internal counter. You just keep calling Sleep() on the class in your tight loop, and every X times it actually calls Sleep(1) instead of Sleep(0). This helps keep responsiveness relatively high in things like spinlocks where we don't expect to be spinning for very long, but also allows lower priority threads run on occasion in case one of those is the one holding the lock (thus avoiding deadlock due to thread starvation).

This kind of "stuttered" sleep, similar to what Hodgman sugguested, is something I will look at. I've done some profiling yesterday and almost 40% of the profiled time is in the Sleep function. But this is for sure a result of the high number of (builder) threads of which I have talked in the posting above (answer to AllEightUp), that is, currently 12 threads on a 4-core CPU.

This sounds very much like the division I use though I kept it generic and called it asynchronous task based processing (aka: Thread Pool with work Queue) and the n-core horizontal distribution system. The problem was always that those two different things tend to fight each other at the most inopportune times. For instance, let's take texture requests, I call them async tasks because they may spend several frames loading data so I don't want them being tied into the main loop distribution system. But, things are never as simple as they seem on the surface, so let me describe the steps involved just so some items are clear:

1. Load (or pull from a metadata cache) the texture header so we have size, mip count etc information. (IO bound or potentially memory bound.)
2. Using #1 information, send a request to the rendering thread to give me an appropriate buffer to store the texture in. The buffer may be a transient chunk of memory the renderer will feed to the Gfx api or it may be the real resource memory block, not up to me to decide on this side of things. (Frame latency bound, i.e. once a frame I pump a message queue and push back results.)
3. Load the actual data. (Io bound again.)
4. For fun, assume it is an 8k x 8k backdrop image and for some reason it is on disk as a jpg, decompress and push it into the buffer. Potentially build mips, maybe even do runtime dxt compression etc. (CPU bound.)
5. Send the 'complete' back to the renderer and fill in a resource handle with a reference to the real resource. (Frame latency and atomic contention bound.)

Well, it seems that we indeed use similar systems. My level terrain texture is 16kx16k large, but will be generated on-the-fly instead of being loaded. It sounds like a task for the GPU, but I mix up 8 different textures per terrain element (8x8 pixel) out of 256 textures, so for the moment it is easier for me to do it on the CPU. Nevertheless, my approach is

1. The rendering systems sends a request to the builder queue/list to fetch a terrain texture chunk.

2. The texture manager controls the texture builders, depending on the camera position, takes a request from the list and send the request to a free builder, which works async.

3. The builder creates a new texture and writes it to a ogl buffer.

4. The texture manager controls the ogl buffer swapping.

5. The builder worker is a simple state machine, the state is controlled by the atomic operations only.

My system, due to the heavy CPU usage, is even more prone to block job-threads

Filling in the empty spaces in the CPU time is exactly why I have the two different solutions and it is also how the OS tends to fulfill various internal items such as async file IO etc. But the nature of the work involved in step #4 (assume it's jpg decompression for example) effectively tricks the OS into doing the wrong things sometimes. It will see the n-core threads as occasional work and prioritize them lower in favor of the async thread doing this big chunk of texture processing. Which means that for the frames that the async task is actively using CPU time, you have effectively lost one of your cores to this thread that the OS won't want to interrupt. What happens is the following (bad ascii art):

1: 11111xxxxxxx1111111xx111xx1111
2: 2222xxxxxxxx2222222244444x2222
3: 3333344444443333333xxxxxxx33xx4444444444
4: aaaaaaaaaaaaaaaaaaaaaaaa....

I.e. Cores are vertical, and threads 1-4 are your n-core per frame workers with 'a' as some busy asynchronous task. The n-core distribution gets fubar'd because thread 4 ends up pushed into the other three 'spurious' threads while 'a' ends up with all of a single core since it is not spurious at this point.

Hmm... This will for sure happen in my case too. Just a thought, wouldn't it help to give the async thread a lower priority and use frequently SwtichToThread or similar to break the run ? Or some global,atomic state which signals, that the job workers are busy, leading to pause the async threads for some time, a load balancer, or would this kill the benefit ?

So, going back to my initial point that you need to look at the top level of things first, the solution for this in my system was more heuristics. I ended up with a solution which at the start of the n-core portion I would maintain the cpu utilization trend and a really simple concept of 'available resources' and then bid things per frame in terms of if I would allow asynchronous tasks to start working. What I mean by this is that I may call a hardware thread .9 units of possible work and add up all the available workers to have say 4 times .9 = 3.6 cpu units available. Based on prior frames, I subtract out n-core usage and any running asynchronous tasks. The remainder is what I use to determine if an asynchronous task will be allowed to start.

So you have some kind of synchronisation between your threads, a load balancing ? Something like this :

if you have for each core a worker and a sister thread for the async work, couldn't these two threads use a simple (virtual) core sharing:

// 1. global state
// only modified by job workers (use atomic operations)
Long virtualCoreSharing[#numberOfWorkerThreads]=0;

// 2. job threads (#cores - 1 + main thread)
// job worker, starting 
virtualCoreSharing[jobWorker.getVirtualCoreIndex()] = 1;

.. process jobs..

// done, give control to main thread
virtualCoreSharing[jobWorker.getVirtualCoreIndex()] = 0;


// 3. texture threads (#cores -1 , no main thread, because the main-thread should be busy all the time)
// execution
for each line in texture do
   // spin lock wait, no while loop to avoid starvation
   if virtualCoreSharing[textureWorker.getVirtualCoreIndex()]==1 do
     // let the sister thread have more CPU time
     Sleep(0); // or higher
   end

   ..process single line or data block
end

I think you're going to get better performance if you pay attention to what jobs you are running in your job pools and when, rather then trying to micromanage where threads run anyway. You generally don't want tasks that write to the same shared data running at the same time to reduce lock contention. (Multiple readers are fine, of course)

That is no issue, I don't have shared data where I write concurrently,everything is either isolated data or double buffered.

This is because context switching between threads on the same core is very expensive

Is this still the case on modern systems ? I've learned that at least on Windows, a lot of performance optimizsation has been done in multithreading. And I'm gettng really jealously when reading all the stuff of how much control you have over consoles...

I decided to profile some of my busy loop stuff and found some interesting results.

All done on Windows 8.1 Pro + Core i7-4770K @ 3.5GHz.

YieldProcessor - 0?s

SwitchToThread - 11?s

SleepEx( 0, TRUE ) - 9?s

SleepEx( 1, TRUE ) - 1790?s

WaitForSingleObjectEx( x, 1, TRUE ) - 1070?s