Linux epoll vs non-blocking sockets
Author
Has anyone tested the scalability of non-blocking sockets each being either read(2)'d or recv(2)'d versus Linux 2.6's epoll(7) interface? Is there a similar interface for Solaris 10/SPARC (6/06)? I am looking for the CPU usage at 100, 1000, and 5000 file descriptors.
Code:
I current have a listening thread and a number of "worker" threads. The listener thread finds new connections, marks them as non-blocking, and handles off the file descriptor the worker thread with the fewest current connections. The worker thread's loop is essentially:
for( number of file descriptors 'i' )
{
int s = filedescriptor
if(read(s, &buffer, size) > 0))
{
/* Verify packet, etc. */
}
}
Hardware:
I am running this on a quad processor UltraSPARC server with 4GB of RAM.
The server does fairly computationally expensive work on each thread, save the listen thread, so single threading is a non-option -- why have four processors if you aren't going to use them?
I think that an interface such as epoll(7) would reduce the total number of userspace->kernelspace transitions as it could test the entire array of file descriptors as the same time, but I have heard on and on that non-blocking I/O + read(2) or recv(2) is the way to go. Considering that I am trying to make a real-time responsive server, and preserve as much CPU time as possible for the requests, is epoll(7) a good idea? Is there a better mechanism under Solaris 10/SPARC (6/06)?
Does anyone have some personal experience with the scalability of such things?
Calling recv() on 5,000 sockets sounds like 5,000 trips into the kernel, which is probably not ideal, unless you know that almost all sockets will have new data almost all of the time.
Just because you have computation in threads, doesn't mean that the receiving needs to be multi-threaded. You can, for example, receive in a single thread, using epoll or similar, and farm out work to the worker threads. I don't know which method would be better on Solaris -- why don't you implement both and measure?
Also, you're not guaranteed to get a whole packet (or only one packet) in the call to read(). Hopefully, you're dealing with partial packets in the "etc" branch.
Just because you have computation in threads, doesn't mean that the receiving needs to be multi-threaded. You can, for example, receive in a single thread, using epoll or similar, and farm out work to the worker threads. I don't know which method would be better on Solaris -- why don't you implement both and measure?
Also, you're not guaranteed to get a whole packet (or only one packet) in the call to read(). Hopefully, you're dealing with partial packets in the "etc" branch.
Author
Farming out the work to other threads seems like a headache from a synchronization standpoint, requiring large amounts of memory being marked as shared, either by setting up a work queue or actually inserting the information to be computed. And it would at best give the same performance that I am getting now, which requires no synchronization in the best case. This of course assumes that the cost of synchronization is greater than the cost of recv(2). While I am not for sure this is true, it seems highly likely, due to the memory traffic involved in copying data around and threads awaiting work and sitting still in between. I was asking more about the scalability of epoll(7) on Linux, and as I found out later, /dev/poll on Solaris.
Yes, I know packets can become fragmented, or rather, not fully recv(2)'d, that was just an overly simplified piece of code.
Please stay on topic: Does epoll(7) under Linux scale better than recv(2) on non-blocking sockets? Any information on Solaris 10/SPARC (6/06)?
Yes, I know packets can become fragmented, or rather, not fully recv(2)'d, that was just an overly simplified piece of code.
Please stay on topic: Does epoll(7) under Linux scale better than recv(2) on non-blocking sockets? Any information on Solaris 10/SPARC (6/06)?
I don't know the answer to your specific question: I suggest you benchmark it yourself.
However, I would like to submit that I disagree with your assessment of what threading means for performance in this case. So that future readers do not get mis-led, my belief is that there is no more copying necessary of data in the threaded case, and synchronization for a thread-specific work queue is typically significantly cheaper than entering the kernel for an I/O primitive like recv().
However, I would like to submit that I disagree with your assessment of what threading means for performance in this case. So that future readers do not get mis-led, my belief is that there is no more copying necessary of data in the threaded case, and synchronization for a thread-specific work queue is typically significantly cheaper than entering the kernel for an I/O primitive like recv().
Quote:Does epoll(7) under Linux scale better than recv(2) on non-blocking sockets?
lol you can't compare epoll with recv on non-blocking sockets. You need all 3 things together.
Author
True, one still does need to use recv(2) or read(2) to actually read the data from the socket. However, what I am refering to is using epoll(7) as a method of determining whether data is present, rather than polling the sockets with a call to recv(2) rapidly and getting an error that data is *not* present.
I think hplus0603 is correct about the userspace->kernelspace transition from a large number of I/O operations probably will use more time merely context switching than getting work done. I am going to benchmark this soon, if anyone would like the results, I can post them here when I am finished.
I think hplus0603 is correct about the userspace->kernelspace transition from a large number of I/O operations probably will use more time merely context switching than getting work done. I am going to benchmark this soon, if anyone would like the results, I can post them here when I am finished.
You typically use EITHER non-blocking sockets OR epoll.
With epoll (or select), you get told about sockets that you can safely recv on without blocking. Thus, you can call recv once, even if the socket is blocking.
With a non-blocking socket, you typically call recv on the socket once through each loop in your game, which is OK for small amounts of sockets (and also for UDP-based servers, that typically only have a single socket), but not for TCP servers with lots of clients.
With epoll (or select), you get told about sockets that you can safely recv on without blocking. Thus, you can call recv once, even if the socket is blocking.
With a non-blocking socket, you typically call recv on the socket once through each loop in your game, which is OK for small amounts of sockets (and also for UDP-based servers, that typically only have a single socket), but not for TCP servers with lots of clients.
Quote:
With epoll (or select), you get told about sockets that you can safely recv on without blocking.
On Winsock newsgroups experts say that this is not true. It can still block.
winsocks is diffrent from sockets winsocks was written as a compatibality layer between windows networking and porting *nix code over to windows
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement