Jump to content

  • Log In with Google      Sign In   
  • Create Account


Linux epoll vs non-blocking sockets


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
13 replies to this topic

#1 Figgles   Members   -  Reputation: 122

Like
0Likes
Like

Posted 10 January 2007 - 02:19 PM

Has anyone tested the scalability of non-blocking sockets each being either read(2)'d or recv(2)'d versus Linux 2.6's epoll(7) interface? Is there a similar interface for Solaris 10/SPARC (6/06)? I am looking for the CPU usage at 100, 1000, and 5000 file descriptors.

Code:
I current have a listening thread and a number of "worker" threads. The listener thread finds new connections, marks them as non-blocking, and handles off the file descriptor the worker thread with the fewest current connections. The worker thread's loop is essentially:

for( number of file descriptors 'i' )
{
int s = filedescriptor[i]
if(read(s, &buffer, size) > 0))
{
/* Verify packet, etc. */
}
}

Hardware:
I am running this on a quad processor UltraSPARC server with 4GB of RAM.

The server does fairly computationally expensive work on each thread, save the listen thread, so single threading is a non-option -- why have four processors if you aren't going to use them?

I think that an interface such as epoll(7) would reduce the total number of userspace->kernelspace transitions as it could test the entire array of file descriptors as the same time, but I have heard on and on that non-blocking I/O + read(2) or recv(2) is the way to go. Considering that I am trying to make a real-time responsive server, and preserve as much CPU time as possible for the requests, is epoll(7) a good idea? Is there a better mechanism under Solaris 10/SPARC (6/06)?

Does anyone have some personal experience with the scalability of such things?

Sponsor:

#2 hplus0603   Moderators   -  Reputation: 4980

Like
0Likes
Like

Posted 11 January 2007 - 11:55 AM

Calling recv() on 5,000 sockets sounds like 5,000 trips into the kernel, which is probably not ideal, unless you know that almost all sockets will have new data almost all of the time.

Just because you have computation in threads, doesn't mean that the receiving needs to be multi-threaded. You can, for example, receive in a single thread, using epoll or similar, and farm out work to the worker threads. I don't know which method would be better on Solaris -- why don't you implement both and measure?

Also, you're not guaranteed to get a whole packet (or only one packet) in the call to read(). Hopefully, you're dealing with partial packets in the "etc" branch.

#3 Figgles   Members   -  Reputation: 122

Like
0Likes
Like

Posted 11 January 2007 - 12:14 PM

Farming out the work to other threads seems like a headache from a synchronization standpoint, requiring large amounts of memory being marked as shared, either by setting up a work queue or actually inserting the information to be computed. And it would at best give the same performance that I am getting now, which requires no synchronization in the best case. This of course assumes that the cost of synchronization is greater than the cost of recv(2). While I am not for sure this is true, it seems highly likely, due to the memory traffic involved in copying data around and threads awaiting work and sitting still in between. I was asking more about the scalability of epoll(7) on Linux, and as I found out later, /dev/poll on Solaris.

Yes, I know packets can become fragmented, or rather, not fully recv(2)'d, that was just an overly simplified piece of code.

Please stay on topic: Does epoll(7) under Linux scale better than recv(2) on non-blocking sockets? Any information on Solaris 10/SPARC (6/06)?


#4 hplus0603   Moderators   -  Reputation: 4980

Like
0Likes
Like

Posted 11 January 2007 - 06:03 PM

I don't know the answer to your specific question: I suggest you benchmark it yourself.

However, I would like to submit that I disagree with your assessment of what threading means for performance in this case. So that future readers do not get mis-led, my belief is that there is no more copying necessary of data in the threaded case, and synchronization for a thread-specific work queue is typically significantly cheaper than entering the kernel for an I/O primitive like recv().


#5 Anonymous Poster_Anonymous Poster_*   Guests   -  Reputation:

0Likes

Posted 21 January 2007 - 07:08 AM

Quote:
Does epoll(7) under Linux scale better than recv(2) on non-blocking sockets?


lol you can't compare epoll with recv on non-blocking sockets. You need all 3 things together.

#6 Figgles   Members   -  Reputation: 122

Like
0Likes
Like

Posted 21 January 2007 - 12:32 PM

True, one still does need to use recv(2) or read(2) to actually read the data from the socket. However, what I am refering to is using epoll(7) as a method of determining whether data is present, rather than polling the sockets with a call to recv(2) rapidly and getting an error that data is *not* present.

I think hplus0603 is correct about the userspace->kernelspace transition from a large number of I/O operations probably will use more time merely context switching than getting work done. I am going to benchmark this soon, if anyone would like the results, I can post them here when I am finished.

#7 hplus0603   Moderators   -  Reputation: 4980

Like
0Likes
Like

Posted 21 January 2007 - 02:41 PM

You typically use EITHER non-blocking sockets OR epoll.

With epoll (or select), you get told about sockets that you can safely recv on without blocking. Thus, you can call recv once, even if the socket is blocking.

With a non-blocking socket, you typically call recv on the socket once through each loop in your game, which is OK for small amounts of sockets (and also for UDP-based servers, that typically only have a single socket), but not for TCP servers with lots of clients.


#8 Anonymous Poster_Anonymous Poster_*   Guests   -  Reputation:

0Likes

Posted 21 January 2007 - 07:48 PM

Quote:

With epoll (or select), you get told about sockets that you can safely recv on without blocking.


On Winsock newsgroups experts say that this is not true. It can still block.

#9 FritzMar   Members   -  Reputation: 140

Like
0Likes
Like

Posted 21 January 2007 - 08:13 PM

winsocks is diffrent from sockets winsocks was written as a compatibality layer between windows networking and porting *nix code over to windows


#10 cobru   Members   -  Reputation: 193

Like
0Likes
Like

Posted 21 January 2007 - 11:01 PM

Have you seen this link ?


#11 hplus0603   Moderators   -  Reputation: 4980

Like
0Likes
Like

Posted 22 January 2007 - 02:32 PM

Quote:
On Winsock newsgroups experts say that this is not true. It can still block.


If select() returns a socket as readable, then recv() will not block on the next read, guaranteed.

This is not true for certain Windows-specific versions, like the infamous WSAAsyncSelect(). Stay away from AsyncSelect().



#12 Anonymous Poster_Anonymous Poster_*   Guests   -  Reputation:

0Likes

Posted 22 January 2007 - 08:21 PM

When 'select' says readable, that means it *was* readable at the time 'select' said it was. It has *never* provided a future guarantee of readability. *No* status reporting system call provides future guarantees.

#13 markr   Crossbones+   -  Reputation: 1653

Like
0Likes
Like

Posted 22 January 2007 - 09:17 PM

You can't compare recv() and epoll they exist for different purposes. epoll is an efficient facility for determining which of a large number of file descriptors has stuff waiting to be read/ written / errors etc.

Of course epoll is vastly preferable to 5000 recv()s if almost all of them return EWOULDBLOCK (e.g. there is no data waiting and they're non-blocking).

But if you know that there are data to read, you may as well read them.

epoll is a Linux-specific facility (although some other Unixes have similar ones) which enables you to add a FD to watch, just once, and continue to watch it until you tell it to stop (or you close it).

Then each call to epoll_wait will wait for any of the things the epoll FD is currently configured to wait for, which could be a great many. This saves the setup work required for select().

Moreover, you can get an array back telling you exactly what has happened on which watched FDs.

Mark

#14 hplus0603   Moderators   -  Reputation: 4980

Like
0Likes
Like

Posted 23 January 2007 - 05:38 PM

Quote:
When 'select' says readable, that means it *was* readable at the time 'select' said it was. It has *never* provided a future guarantee of readability. *No* status reporting system call provides future guarantees.


Not true. Once data is queued for the incoming socket, then that file descriptor WILL be readable, until you close that socket. If select() returns a given socket as readable, it WILL NOT BLOCK the next time you call recv(), unless you take explicit action on that socket in the meanwhile.

There may be other kinds of file descriptors (not sockets) where this is not true. However, the interaction between select/recv() is defined to be exactly this, for good reason. Also note that I'm talking about recv(), not read().




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS