latency, TCP_QUICKACK, etc.
I'm about to start diagnosing a few latency issues with one server app - basically it seems like it can only receive 4 or 5 small TCP messages a second. (This may be a Windows-only measurement, but I'm not sure right now.) The sending socket has TCP_NODELAY set, which I believe stops buffering on the local side, but I'm guessing the issue comes from the lack of ACKs from the recipient. Theoretical latency should be <1ms as this is over a LAN.
First question: Would I be right in guessing that setting TCP_QUICKACK on the receiving socket will help here? (The final server should be on Linux so portability isn't a great concern.)
Second question: Would it be equally good to simply send back a tiny packet such as 'OK'? That would need explicitly reading and discarding at the application level, but perhaps is more predictable than setting non-portable socket options.
Third question: how do TCP buffer lengths factor into this, on both sides? Are there some optimal settings that minimise latency at the expense of bandwidth, which is cheap in this particular case?
Are you sure this isn't your application's fault?
TCP does its own ACKs, so sending back something isn't necessary. If some particular data doesn't arrive, the connection will timeout. TCP_NODELAY should send data immediately, unless there's something fishy with router/windows/network/... going on.
But TCP ACKs are automatic.
TCP does its own ACKs, so sending back something isn't necessary. If some particular data doesn't arrive, the connection will timeout. TCP_NODELAY should send data immediately, unless there's something fishy with router/windows/network/... going on.
But TCP ACKs are automatic.
TCP ACKs are automatic, but they're not instant. If there's data due to go back in that direction, they'll be sent as part of that packet, but if there is no data going back, then the socket waits for a short period of time to see if such a candidate packet appears, to avoid wasting bandwidth on nearly empty packets. Only after that timeout will it then send such a packet with nothing except an ACK (and presumably sequence numbers, etc). I believe on Windows this timeout period is always 200ms, and on Linux the timeout varies according to the estimated round trip time, but the timeout always exists unless you turn it off with TCP_QUICKACK on Linux or via some sort of registry setting on Windows.
You can't even see TCP acks from the client side. The TCP ack opens the sliding window; that's all. After the connection is established, the window should be plenty open already.
How are you measuring "five packets a second"? Are you running client and server on the same machine, perhaps? Then you might be seeing scheduler effects, rather than networking effects.
If it turns out you're sending more data than the TCP window can reasonably sustain (unlikely for a game on a LAN), then the QUICKACK extension might help, if you have it. Somehow, I don't think that's the problem you're seeing, though.
How are you measuring "five packets a second"? Are you running client and server on the same machine, perhaps? Then you might be seeing scheduler effects, rather than networking effects.
If it turns out you're sending more data than the TCP window can reasonably sustain (unlikely for a game on a LAN), then the QUICKACK extension might help, if you have it. Somehow, I don't think that's the problem you're seeing, though.
I'm not sure what you mean by "you can't see TCP acks from the client side". I know they aren't visible to application code but there are circumstances where a send won't take place until an ACK has come through. One source tells me that this is only the case when there's no send buffer and TCP_NODELAY hasn't been set, but anecdotally there seems to be more to it than that.
We're not sending too much data, just encountering too much latency when sending lots of very small amounts of data unidirectionally. I was led to understand QUICKACK was of relevance here, but there are a lot of factors involved which I don't fully understand yet.
We're not sending too much data, just encountering too much latency when sending lots of very small amounts of data unidirectionally. I was led to understand QUICKACK was of relevance here, but there are a lot of factors involved which I don't fully understand yet.
Quote:there are circumstances where a send won't take place until an ACK has come through
You didn't quite answer how you are measuring the latency.
Anyway, TCP works by having an open "window," which is how much the sending side will put on the wire without getting an acknowledge. The ACK packets will remove from the amount of data on the wire, thus making space for the sender to send more data. This is known as the "sliding window" approach.
Thus, if you fill the entire window (which isn't terribly big -- on the order of 64 kiB) then you can't send any more until you get an ack from the other end. However, if you're sending "small" packets of data, I would assume that you're not filling the window. The receiving side may delay acks, but only until it has half of the outstanding window deferred, at which point it should send an ack.
You could experiment by increasing the buffer sizes before you establish the connection; this will allow the TCP implementations to use window scaling to make the window bigger than 64 kiB. Try setting the buffer size to 1024*1024 on each side, before calling connect() or accept(), and see what happens.
I don't have an answer regarding the latency issue right now unfortunately. I'll try and take a look at it today to work out what's going on. From what I've learned here and elsewhere, I'm under the impression that having Nagle disabled and a large enough send buffer should allow us to send each small message immediately. I'm told that it's not working out quite like this, but we'll see. This thread really is just me trying to get an idea of what could be going on and what potential factors I may have to look into.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
