How to Design for Threading
Author
I'm not sure how this works so I'll paint a hypothetical picture to stand-in as a good example...
Say I am designing a web app that will contact a server-side DLL (C# .NET assembly) when a user does something (e.g., clicks a button). The client sends some form of an XML message that the serverside DLL - aka "Widget.dll" - reads in as data and processes.
The problem here is that there could be tens of thousands of simultaneous users at any given time. I am looking for elegant solutions that (will most likely involve multi-threading and) guarantee my Widget.dll doesn't get bogged down with all the traffic and trying to process all these XML requests.
I have become familiar with System.Threading and have some questions:
(1) Is there an upper-limit to the number of threads one can/should open for a specific instance of an object? I can't seem to find any docs/ref material on this?
(2) Are there any algorithms for calculating the number of threads that should be opened at any given time, or more generally, any algs that would be applicable and useful to this problem (from a resource mgmt perspective)?
(3) Are there other elegant ways of acquiescing massively-multiple users? For instance, perhaps having multiple servers, and/or multiple Widget.dll's running on each server?
thx,
ply
Ideally, the number of threads will be closer to the number of cores than to the number of users/connections/whatever. For instance, you can set up a listening thread that receives incoming requests and pushes them onto a queue, then use one worker thread per core to extract requests from the queue and serve them. This is because if you have four cores, you can only serve four requests at a time, and any attempt to serve more will just slow everything down.
The only special case is when your worker threads have to wait on an asynchronous event (such as querying another server). In that case, you can have several threads per core in order to make use of the free time while other threads are waiting.
The only special case is when your worker threads have to wait on an asynchronous event (such as querying another server). In that case, you can have several threads per core in order to make use of the free time while other threads are waiting.
The general rules are:
Keep the number of threads to a minimum, as it takes significant time to swap threads.
Keep the number of CPU bound threads equal to the number of logical threads your processor supports.
For MEMORY bound threads it can be tricky, since logical threads your processor supports can be higher than the number of cores / cache banks. This means you want AT MOST a number of threads equal to the number of cores for some setups to avoid cache thrashing.
You can have many IO bound threads, but keep the number as low as possible.
What you probably want is a thread pool of CPU bound workers to process your requests, and then one thread to recieve the requests and enqueue the data for the processing pool.
--addendum.
A dual core xeon with hyperthreading is 1 processor die, 2 processor cores, and 4 logical threads.
Some spark servers support hundreds of logical threads per core. Other processors only support one logical thread per core.
Keep the number of threads to a minimum, as it takes significant time to swap threads.
Keep the number of CPU bound threads equal to the number of logical threads your processor supports.
For MEMORY bound threads it can be tricky, since logical threads your processor supports can be higher than the number of cores / cache banks. This means you want AT MOST a number of threads equal to the number of cores for some setups to avoid cache thrashing.
You can have many IO bound threads, but keep the number as low as possible.
What you probably want is a thread pool of CPU bound workers to process your requests, and then one thread to recieve the requests and enqueue the data for the processing pool.
--addendum.
A dual core xeon with hyperthreading is 1 processor die, 2 processor cores, and 4 logical threads.
Some spark servers support hundreds of logical threads per core. Other processors only support one logical thread per core.
It seems you're using some web app framework, and you merely provide a plug-in dll.
In that case, threading will be performed by server as needed.
Even without it, there's little need for threading for purposes of scalability. Only exception would be blocking calls in third-party API (SQL for example), in which case running those in separate thread pool of workers may help. Choosing a non-blocking API however is likely to be considerably more scalable.
Main reason for this is that this type of plugins is usually request centric, giving you little room for global management of resources. This simply means you'd need to create/destroy threads for each request you receive which is a disaster.
In that case, threading will be performed by server as needed.
Even without it, there's little need for threading for purposes of scalability. Only exception would be blocking calls in third-party API (SQL for example), in which case running those in separate thread pool of workers may help. Choosing a non-blocking API however is likely to be considerably more scalable.
Main reason for this is that this type of plugins is usually request centric, giving you little room for global management of resources. This simply means you'd need to create/destroy threads for each request you receive which is a disaster.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
