Sign in to follow this  

data recv-ed in bundles

This topic is 4130 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi, i am having trouble sending/recieving small amounts of data at a time, using winsocket TCP/IP in c++. My server sends 300 to 1500 bytes of data every frame (in one send call), 60 frames per second. Everything is recieved by the client (a seperate thread using recv, recieving data as soon as possible) but it arrives in bundles of about 5 frames at a time. I tried the following things to fix it but i noticed no real improvement: - calling recv to read data sizes like 32, 100 or 300 bytes at a time (32 gave a slightly better performance than 100 or 300) - enabling TCP_NODELAY (server and client) - sending more data (if for some reason TCP_NODELAY didn't work) - decreasing the read/send buffer size - sending the data in smaller parts (50 bytes) - changing the priority class of the reading thread What else can i try, what could be the problem?

Share this post


Link to post
Share on other sites
Are you complaining about lag recieving data or about the bundling? You've done the usual stuff about TCP lag so I don't really have anything else to suggest.

As for the bundling issue you do realize that there is no guarantee with TCP that just becuase you sent 300 bytes in a single recv that you will get 300 bytes on the other end? It's perfectly ok as far as TCP is concerned for it to give you your data one byte at a time, or 15000 bytes at time (i.e. bundling), or anything in between. There is no such thing as a packet at the application layer when using TCP.

Share this post


Link to post
Share on other sites
Look at the forum faq,

Quote:

Q14) I'm trying to send two packets over TCP, but it seems like they arrive "merged" or one packet disappears on the receiving end. What's going on?


This is normal and expected behaviour.

Enabling TCP_NODELAY on the sender will usually stop the joining breaking messages up (unless its send queue is nonempty already, usually indicating some congestion, or you send messages *really* fast), but won't stop the receiver doing so.

Sending 60 times per second is too often for most purposes. You will find coalescence happens. Sending 300 bytes 60 times per second means 18 kbytes per second, or 144 kbit/second, which is faster than many peoples' connections. This is too fast.

In any case, even if message coalescence does not happen under normal circumstances, your application should be able to handle it anyway; as the FAQ indicates, TCP does NOT preserve message boundaries.

Mark

Share this post


Link to post
Share on other sites
I'm complaining about the bundling, it causes multiple frames to arrive at the same time, which effectively gives you a framerate of at most 10 fps (even though there are 60 frames sent every second, the older ones just get overwritten by the new ones recieved in the same bundle).

When TCP_NODELAY is on (so nagle is turned off) data should be sent immediately (so not wait for more data to make a nice big package), there is no reason for the network adapter (or the localhost pipe, which has the same problem at the moment) to hold on to it longer than that.

So either nagling isn't turned off at all (while getsockopt says it is), or for some reason recv() only recieves data when some insanely large buffer is completely full or after a timeout of something like 200 ms, which sounds like nagle to me.

Any ideas?

EDIT:

Quote:
Original post by markr
Enabling TCP_NODELAY on the sender will usually stop the joining breaking messages up (unless its send queue is nonempty already, usually indicating some congestion, or you send messages *really* fast), but won't stop the receiver doing so. ...


Yeah, but for some reason it does not right now, even when i use localhost it still gets bundled (which rules out congestion i'd say).

Is there any other way to prevent this? I doubt that all TCP traffic has a maximum of 10 blocks of data every second, so why doesn't it send (or recieve) them right away?

Share this post


Link to post
Share on other sites
Is the "bundling" you've observed actually resulting in larger packets? Do a packet capture (with Ethereal or something) and see what's actually in the TCP packets (which is different than what recv() tells you). It's possible that the buffering is the fault of the receiving thread not receiving often enough. If nothing else, packet tracing will allow you to localize the problem to the sender or the receiver.

Share this post


Link to post
Share on other sites
Okay, i tried Ethereal and it turns out the packets are being sent and recieved as individual packets but the timing is all screwed up.

They are sent with a few packets at a time, so with timing like this:

50.1650..
50.1668..
50.1686..
50.2290..
50.2319..
50.2337..
50.2920..
50.2938..
50.2954..

I guess that's because there's a seperate thread in the winsocket drivers that actually writes the data to the network adapter. Because of windows's time sharing it get's the time to do so roughly 15 times a second.

That would mean the only way to sort of fix my problem is to smear out the recieved data and make it look a little smoother. Or is there a way to make winsock truly send the data immediately when i call send()?

Or, have i got it all wrong :P?

Share this post


Link to post
Share on other sites
None of your threads are busy-waiting are they? If so, nothing will work as expected as you'll get client/server contention when you don't want it.

Have you tried running the client and server on separate hosts?

Mark

Share this post


Link to post
Share on other sites
I really think you're coming at this whole problem wrong. TCP is just not designed to do what you seem to want it to do. You can tweak things until you're blue in the face and you may even get it to work most of the time in extremely controlled scenarios but as soon as you put in on a different machine, or a different network, or somebody external to your machine starts sends you data then you'll change the timing and it won't work anymore.

You need to code your app to handle this sort of thing, not tweak things to try to get it not to happen.

Share this post


Link to post
Share on other sites
Quote:
Original post by markr
None of your threads are busy-waiting are they? If so, nothing will work as expected as you'll get client/server contention when you don't want it.

Have you tried running the client and server on separate hosts?


Yes, that gives me the same problem. And the server thread that sends the data is the same thread that does the physics, which does appear smooth when i let the server view it, so it's somewhere between the send() and recv() command that it goes wrong.

I've added a timecode to the data packets, smoothing it out over several frames and it now works fine (just with a small lag).

I guess i should have gone with udp for this sort of thing.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by Tree Penguin
Quote:
Original post by markr
None of your threads are busy-waiting are they? If so, nothing will work as expected as you'll get client/server contention when you don't want it.

Have you tried running the client and server on separate hosts?


Yes, that gives me the same problem. And the server thread that sends the data is the same thread that does the physics, which does appear smooth when i let the server view it, so it's somewhere between the send() and recv() command that it goes wrong.

I've added a timecode to the data packets, smoothing it out over several frames and it now works fine (just with a small lag).

I guess i should have gone with udp for this sort of thing.


When data arrives on a tcp connection, the receiving side buffers it up until its buffer is full. When the receiving process doesn't read the data out from the buffer before the next packet arrives, the system simply appends the new data to the end of the buffer. If you want to get 60 frames per second by sending 60 packets per second, you have to make sure that both the sending and the receiving side gets a timeslice at least 60 times per second. On systems that have a timeslice larger than 1/60 seconds it's impossible to do it. On systems with a timeslice exactly 1/60 seconds, the only time this will work is when the only process that's ready for execution is the sending or the receiving process. On systems with a timeslice smaller than 1/60 this will work as long as the total number of timeslices per second divided by 60 is larger than the number of processes ready for execution. Various priority levels have to checked too, since they modify the scheduling.

Generally, you either lower the network framerate and interpolate/exrapolate or get an os that can provide realtime scheduling. The first seems easier.

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous Poster... If you want to get 60 frames per second by sending 60 packets per second, you have to make sure that both the sending and the receiving side gets a timeslice at least 60 times per second. On systems that have a timeslice larger than 1/60 seconds it's impossible to do it.

They get their time, at least the sending side does. I looked at the exact times i called send (which where fine) and the exact times the data actually got sent (which where screwed up), so the winsock driver itself doesn't get or take the time.

Quote:
It's probably not too late to just switch to UDP.

Wouldn't switching to UDP mean i need to change the whole logic of my server-client code? I got one evening (4 ours or so) to switch, is that enough for an average programmer?

Share this post


Link to post
Share on other sites
4 hours is not enough to do anything when it comes to distributed programming, unfortunately.

You could build a simple "reliable" layer on top of UDP, and replace your TCP connect/send/recv calls with calls to that layer. However, building and testing that layer probably takes more than 4 hours. Even integrating an existing layer, like SDL_Net, HawkNL or ENet, might take that long.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
4 hours is not enough to do anything when it comes to distributed programming, unfortunately.

You could build a simple "reliable" layer on top of UDP, and replace your TCP connect/send/recv calls with calls to that layer. However, building and testing that layer probably takes more than 4 hours. Even integrating an existing layer, like SDL_Net, HawkNL or ENet, might take that long.


Too bad, thanks anyway.

Share this post


Link to post
Share on other sites
If the delay is the tcp window buffering, couldnt he just turn off scaling and force a small window (eg mtu = 500, windowsize=500)? If the delay is the QoS bucket (I'm not sure what MS call it or even if there is one by default) then maybe he can fiddle that too.

Share this post


Link to post
Share on other sites

This topic is 4130 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this