Too many threads in this server application?
Author
I haven't run into an issue yet, but I came to realize that my current design structure for a server involves a lot of threading - far more than there should probably be - and that being a server, I cannot afford large performace losses, especially when under stress.
At present, there is a thread that handles input from the Host User (ie, one that enables the administrator to break in and change things [they are mutex/semaphore safe, however] by submitting commands at any time), a thread that handles the main operations of the server (updating all there is to be updated, and once again mutex/semaphored), and finally a thread for each active connection, which processes all messages from a specific socket (and yes, semaphores and mutexes keep them from writing to the same location at once).
Now, it sounds like there are just 3 threads (I don't know if that's a lot already or not) but once there are many users (in a beautiful world of never-going-to-happen-but-I-can-dream-anyway, 200 or more realistically, 10) there are that many more threads running and processing data. That sounds to me like a lot at once. I am inexperienced with threads, so I don't know what a realistic number is. I can say that these threads aren't "working hard," but they are working and it seems as though working "simultaneously" is the right way to go (for a server).
Would it be better to combine the socket threads to a single thread, thus reducing the total thread count to 3? This means that each individual connection would have to wait for all others, but the penalties involved there are perfectly acceptable if the overhead for running them "simultaneously" (I'll ignore time splicing and use this word) would be greater than running them separately.
One fear is, though, that these sockets are synchronous. If I combine the threads, I'll have no choice but to make them asynchronous so a non-responding socket wont jam the whole stack.
I realize, of course, that this differs from machine to machine. In all actuality, though, the machine wont be great.
Anyway, thank you for any assistance, I appreciate any responses.
Note: I didn't mention my OS - though it's probably significant - because I'm using SDL for platform independence. Considering that, what is the best solution?
Yes, asynchronous sockets running on a single thread is the only thing that scales. A thread per socket is fine if you have 5-15 clients or so. Above that, it just gets silly.
Quote:Note: I didn't mention my OS - though it's probably significant - because I'm using SDL for platform independence. Considering that, what is the best solution?
select() over non-blocking sockets is only semi-portable way.
Otherwise, depending on OS, there's IOCP, epoll, aio or kqueue.
Finally, there's several portable libraries, such as libevent or boost's asio.
Quote:Would it be better to combine the socket threads to a single thread, thus reducing the total thread count to 3?
Unless you're dealing with upwards of tens of thousands of connections, there's no need for more than one networking thread.
Might I suggest you read up on the use of thread pools.
The general idea is that you have a single dispatcher thread that waits for read activity on all connected sockets. When a read-available condition is detected (using select(), epoll(), or whataver is the gold standard on your OS), it queues a notification. A pool of (one or more, the number may even be dynamically configurable at runtime) worker threads is waiting on that queue, and the first available free thread grabs the notice, reads from the socket, and peforms the procesisng then goes back to wait for more work.
If a socket gets "stuck" only one thread is taken out.
The general idea is that you have a single dispatcher thread that waits for read activity on all connected sockets. When a read-available condition is detected (using select(), epoll(), or whataver is the gold standard on your OS), it queues a notification. A pool of (one or more, the number may even be dynamically configurable at runtime) worker threads is waiting on that queue, and the first available free thread grabs the notice, reads from the socket, and peforms the procesisng then goes back to wait for more work.
If a socket gets "stuck" only one thread is taken out.
Quote:Original post by Spoonbender
Yes, asynchronous sockets running on a single thread is the only thing that scales.
That is just not true. A single thread using asynchronous sockets scales well on a single-core server but falls behind a well-designed threaded implementation on today's architectures. It's pretty hard to find server hardware with only a single CPU core these days.
It's true that the thread-per-socket scales very poorly and should be avoided.
A design that uses a thread count that is some small multiple of the core count and lock-free programming scales much much better than a single asynchronous thread, and is far easier to write and maintain.
Quote:Original post by Zouflain
and finally a thread for each active connection
Never do that. "The "one thread per client" model is well-known not to scale beyond a dozen clients or so." Actually, that's a bit of an exaggeration, as I've had hundreds of simultaneous connections with 2 threads per connection, but it does indeed thrash your server.
Quote:One fear is, though, that these sockets are synchronous. If I combine the threads, I'll have no choice but to make them asynchronous so a non-responding socket wont jam the whole stack.
Did you start with Java, by any chance?
Quote:Original post by KylotanQuote:Original post by Zouflain
and finally a thread for each active connection
Never do that. "The "one thread per client" model is well-known not to scale beyond a dozen clients or so." Actually, that's a bit of an exaggeration, as I've had hundreds of simultaneous connections with 2 threads per connection, but it does indeed thrash your server.
Actually, I've been reading a lot of papers on RPC, with quite a few talking about server scalability. It turns out on modern operating systems (like Linux since NPTL was introduced), one thread per connection turns out to perform a lot better than using any form of poll/select. Which is actually reflected in my own performance tests. There are papers that talk about how Java's original IO is better than NIO for that very reason.
Quote:Original post by abdulla
Actually, I've been reading a lot of papers on RPC, with quite a few talking about server scalability. It turns out on modern operating systems (like Linux since NPTL was introduced), one thread per connection turns out to perform a lot better than using any form of poll/select. Which is actually reflected in my own performance tests. There are papers that talk about how Java's original IO is better than NIO for that very reason.
Which is somewhat interesting, given that JVM doesn't have network layer, and all calls are passed down to OS.
How well does it work with 5000 concurrent connections?
Any chance of a link to the articles?
I have worked with a service that has 3 threads per certain piece of external hardware, of which there can be more than 600 of, that's over 1800 threads.
We started to have problems with running out of virtual address space and had to cut down the amount of stack per thread.
If we were to do a rewrite then we would use some kind of thread pooling mechanism.
We started to have problems with running out of virtual address space and had to cut down the amount of stack per thread.
If we were to do a rewrite then we would use some kind of thread pooling mechanism.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
