Sign in to follow this  

Server call to non-blocking send succeds but client don't receive packet

Recommended Posts

Posted (edited)

Hi everyone,

I'm building my mmorpg from scratch, it's going pretty well I would say. ( I will surely open a development blog soon)

My network protocol is built on top of tcp, and a couple of days ago I implemented a way for me to spam other players in the world, that open their own connection to the server. (for now I'm running both client and server on localhost).

Everything is fine, (indeed I'm surprised to see 300 players easily handled in these conditions) except for a sublte bug that I'm having from yesterday.

 

My world is subdivided in islands, and when a player changes island, the server sends the player information about the new island, so that the client can open a new connection to the other server. (Yes I'm running 2 server on my pc)

When the numbers of players increase, sometimes the client don't receive that "critical" message, and so the connection to the other island never happen, even if the server has sended it. (or at least the call to send() returned the right number of bytes).

 

So my question is: could it be the case that even if the call to send() effectively returns the right number of bytes, in reality the clients dont receive them because their buffer is too full? that would be explained by the fact that all is running locally, so I guess both client and server are sharing the same tcp receiving buffers at their base.

Shound I implement an "acknowledge" for those critical packet? so that until the server don't receive the acknowledge it continues to send the critical packet again and again?

That would sound a bit odd, because that's the principle around tcp right? but I'm pretty sure the bug is not something in the logic of the game, is just that the server call to send return a "fake" value.

Maybe in real world scenario this will never happen? 

 

Thank you all, Leonardo.

Edited by erpeo93

Share this post


Link to post
Share on other sites
Posted (edited)

"No indication of failure to deliver is implicit in a send()."

Locally detected errors are indicated by a return value of -1.

 

that's on the man page, so I guess I was right... the client isnt really receiving the bytes.

BUT, at this point I'm asking, considering this fact, How can my server be sure that the packet has been sended? the only way is to wait for an acknowledge right? but now I'm implementing a reliable protocol by myself, which is exactly the reason I used tcp over udp...

Edited by erpeo93

Share this post


Link to post
Share on other sites

How can my server be sure that the packet has been sended? the only way is to wait for an acknowledge right? but now I'm implementing a reliable protocol by myself, which is exactly the reason I used tcp over udp


If you need exactly the guarantees that TCP gives -- every byte is received, in the order it was enqueued -- then using TCP is great!
Note that TCP doesn't GUARANTEE that the data gets there -- it gets there, or you (eventually) get a timeout error. I mean, if the cable is cut or the computer turned off, no data will actually make it there.

Most game traffic can work better when the guarantees are different, though. For example, why re-transmit old object state, when you have newer object state available? And if each object update is un-ordered compared to each other, why delay the update for object B just because the update for object A was somehow lost?
It is in these cases that you CAN do better on UDP than TCP. But if that's not important to you, TCP can be just fine.

Share this post


Link to post
Share on other sites

"could it be the case that even if the call to send() effectively returns the right number of bytes, in reality the clients dont receive them because their buffer is too full?" - Yes. Confirmation of sending is not proof of receipt. When it says "Locally detected errors are indicated by a return value of -1" that 'locally' doesn't mean "on my machine", it means "in the sending process".

"that would be explained by the fact that all is running locally, so I guess both client and server are sharing the same tcp receiving buffers at their base." - No, this isn't necessarily the reason, nor likely to be true anyway. One possibility is that you aren't reading the data on the client as quickly as you need to, in order to keep the buffer empty. But that would normally mean the connection gets dropped soon after.

"Should I implement an "acknowledge" for those critical packet?" - No, but you must be checking for errors everywhere in your networking code. It sounds more likely that there is some other problem that means this message is going missing. Does that client receive subsequent messages? Do subsequent messages to other clients work okay? Are you sure you're reading data quickly enough on the clients?

Share this post


Link to post
Share on other sites

 

How can my server be sure that the packet has been sended? the only way is to wait for an acknowledge right? but now I'm implementing a reliable protocol by myself, which is exactly the reason I used tcp over udp


If you need exactly the guarantees that TCP gives -- every byte is received, in the order it was enqueued -- then using TCP is great!
Note that TCP doesn't GUARANTEE that the data gets there -- it gets there, or you (eventually) get a timeout error. I mean, if the cable is cut or the computer turned off, no data will actually make it there.

Most game traffic can work better when the guarantees are different, though. For example, why re-transmit old object state, when you have newer object state available? And if each object update is un-ordered compared to each other, why delay the update for object B just because the update for object A was somehow lost?
It is in these cases that you CAN do better on UDP than TCP. But if that's not important to you, TCP can be just fine.

 

Yeah, I totally agree with you on the old object-state thing.

Yesterday I implemented acks for the critical packets, and of course it worked.

So I guess that at this point tcp is basically useless to me, the ideal protocol for my game is clearly one that sits on top of udp for the client-server communication.

 

For my server-server communications, although, I think I will stick with tcp: I know that my servers will be always very near one to the another, so there wont be performance problems or overhead in the tcp ack mechanism. And of course there are more critical packets server-server than client-server, it would be hard to mantain all of them in memory.

 

It's safe to use udp for client-server communications while using tcp for server-server communications?

Moreover, IF I know that the host of the tcp communication are very near and will always have room to receive the packets, can I assume that if send return me the right number, the packet will be received by the destination?

Share this post


Link to post
Share on other sites

clients dont receive them because their buffer is too full

That's unlikely. While it is true that clients will not receive data as long as the buffer is full (but then, just read more often!), as soon as the buffer has room again, clients will receive the data from resent datagrams. Unless... the server crashed in the mean time or someone pulled a cable, or several minutes passed with no sign of life from the client whatsoever.
 

non-blocking send

Here, there is indeed a possibilitiy that clients are not getting the message, that is if it exceeds the send buffer size. Normally this is harmless, the call to send will just block and will copy data in smaller chunks until none is left, then eventually return. But with non-blocking send, you get EWOULDBLOCK and that's it. If you don't check for send being successful, it just goes into oblivion and you never know. Be sure you do check the return code and errno.

 

the only way is to wait for an acknowledge

Yes, that is the only way of knowing the client has gotten the message, but the approach makes little sense. You already know for certain that unless catastrophic things happen (pull cable, crash, etc) the client will receive the message. You also know that data is delivered in-order, i.e. any message referring to what happens on the other island will be received after the "move to new island" message is received. Both things are guaranteed by TCP. No issue with consistency there.

Thus, resending does not make any sense. If the connection is broken, the resend will not make it through either, but if there is any way of getting the data to the client, it will be received. And, it will be received before anything you send after it. Your manual resend won't, it will be received after whatever you sent last (so it isn't terribly useful for maintaining consistency).

TCP can be tricky if one wants some particular things which are not directly supported by either the protocol or the API. You cannot tell how much the client received (not easily and portably, anyway), nor whether the connection is alive at all, other than by sending and getting a connection reset or destination unreachable error (even if you get no error, that still doesn't mean much since the condition may change the next instant), or by closing the socket and checking the return value. Neither one is particularly useful in practice(especially closing the socket is not if you're intending to continue sending!).

Luckily, almost every time when you think you need such a functionality, you really don't. Just send your stuff and rely that it will work. Your sends might not arrive with the lowest possible latency, but they will arrive, and in-order. Or, well, sometimes they won't, but then you will eventually get an error, and there's not much you could do to remedy the problem anyway. -- close socket, and wait for the client to connect again.

Or, as hplus0603 suggested, consider UDP instead. UDP sounds very scary because it is unreliable. But the truth is, it isn't unreliable at all -- the datagrams arrive just as reliably as any other packet, 99.9% of the time it just works. Only, UDP doesn't strictly guarantee an order, nor does it resend packets in those 0.01% of cases where they get dropped -- but in a realtime application like a game (or think telephony!) this is often not something you would want anyway (who cares about what was 5-10 seconds ago?). But if you do care because one datagram was particularly important, you can still just resend the datagram.

Share this post


Link to post
Share on other sites

"could it be the case that even if the call to send() effectively returns the right number of bytes, in reality the clients dont receive them because their buffer is too full?" - Yes. Confirmation of sending is not proof of receipt. When it says "Locally detected errors are indicated by a return value of -1" that 'locally' doesn't mean "on my machine", it means "in the sending process".

"that would be explained by the fact that all is running locally, so I guess both client and server are sharing the same tcp receiving buffers at their base." - No, this isn't necessarily the reason, nor likely to be true anyway. One possibility is that you aren't reading the data on the client as quickly as you need to, in order to keep the buffer empty. But that would normally mean the connection gets dropped soon after.

"Should I implement an "acknowledge" for those critical packet?" - No, but you must be checking for errors everywhere in your networking code. It sounds more likely that there is some other problem that means this message is going missing. Does that client receive subsequent messages? Do subsequent messages to other clients work okay? Are you sure you're reading data quickly enough on the clients?

Yeah, at one point probably the client is flooded with packet from the server (In some moments the client was receiving updates for maybe 100 players, and so the buffer gets filled out very quickly... in fact the bug never happens until I spawn a big number of players.

I'm not sure that I'm reading data quickly enough when there are 100 players in a small area (probably not), but how can I determine if it so? And if it is so, what can I do?

Share this post


Link to post
Share on other sites

The simplest thing is to keep reading until you know the socket has no more data to give you. If you end up in an infinite read loop, you're not reading as fast as you're sending. Another common occurrence is that you end up spending something like 99% of your time reading and barely any time doing other processing, until eventually the connection gets dropped as data backs up.

Share this post


Link to post
Share on other sites
Posted (edited)

 

clients dont receive them because their buffer is too full

That's unlikely. While it is true that clients will not receive data as long as the buffer is full (but then, just read more often!), as soon as the buffer has room again, clients will receive the data from resent datagrams. Unless... the server crashed in the mean time or someone pulled a cable, or several minutes passed with no sign of life from the client whatsoever.
 

non-blocking send

Here, there is indeed a possibilitiy that clients are not getting the message, that is if it exceeds the send buffer size. Normally this is harmless, the call to send will just block and will copy data in smaller chunks until none is left, then eventually return. But with non-blocking send, you get EWOULDBLOCK and that's it. If you don't check for send being successful, it just goes into oblivion and you never know. Be sure you do check the return code and errno.

 

the only way is to wait for an acknowledge

Yes, that is the only way of knowing the client has gotten the message, but the approach makes little sense. You already know for certain that unless catastrophic things happen (pull cable, crash, etc) the client will receive the message. You also know that data is delivered in-order, i.e. any message referring to what happens on the other island will be received after the "move to new island" message is received. Both things are guaranteed by TCP. No issue with consistency there.

Thus, resending does not make any sense. If the connection is broken, the resend will not make it through either, but if there is any way of getting the data to the client, it will be received. And, it will be received before anything you send after it. Your manual resend won't, it will be received after whatever you sent last (so it isn't terribly useful for maintaining consistency).

TCP can be tricky if one wants some particular things which are not directly supported by either the protocol or the API. You cannot tell how much the client received (not easily and portably, anyway), nor whether the connection is alive at all, other than by sending and getting a connection reset or destination unreachable error (even if you get no error, that still doesn't mean much since the condition may change the next instant), or by closing the socket and checking the return value. Neither one is particularly useful in practice(especially closing the socket is not if you're intending to continue sending!).

Luckily, almost every time when you think you need such a functionality, you really don't. Just send your stuff and rely that it will work. Your sends might not arrive with the lowest possible latency, but they will arrive, and in-order. Or, well, sometimes they won't, but then you will eventually get an error, and there's not much you could do to remedy the problem anyway. -- close socket, and wait for the client to connect again.

Or, as hplus0603 suggested, consider UDP instead. UDP sounds very scary because it is unreliable. But the truth is, it isn't unreliable at all -- the datagrams arrive just as reliably as any other packet, 99.9% of the time it just works. Only, UDP doesn't strictly guarantee an order, nor does it resend packets in those 0.01% of cases where they get dropped -- but in a realtime application like a game (or think telephony!) this is often not something you would want anyway (who cares about what was 5-10 seconds ago?). But if you do care because one datagram was particularly important, you can still just resend the datagram.

 

very useful reply, thank you.

I'm absolutely considering udp at this point, I'm just waiting for me to have anything more compelling to do on the game that to change a protocol that is working at the moment.

 

So, to say that one last time, IF send() returns me the right number (the total size of the packet), then the packet WILL arrive at destination (unless catastrophic events of course) and it will arrive before anything other that I send after this send, is that right?

 

If it's so, then it means that the client is closing the connection at some point, for a reason that I don't know at the moment... because the server of course keeps running and pushing packets all over the place.

 

 

Noooo, I know the reason... I spotted the bug in this exact moment...

After successfully sending the critical message, the server calls shutdown on the client socket, and then set a boolean that is "loggingOut"

Then, after having done all of the network stuff, the server checks if for every player the loggingOut variable it's true, and if it is so it calls closesocket.

Of course, when there is a snall number of players, the call to closesocket is NEVER done before the packet is actually delivered, because the receiving buffer isn't full.

But when the receiving buffer is full, even if the call to send succeds, the packet hasn't been delivered, but the variable loggingOut is setted to true anyway... and here we go closesocket is called.

What should I do? should I set a timer? wait for a special message?

 

Man, that was nasty. 

 

But of course now I've implemented that useless acking thing, that will be exactly what I would do if I was using UDP... I will keep it off hand so that I have it ready when it comes the time to switch to udp.

what do you think about using udp for client-server and tcp for server-server?

Edited by erpeo93

Share this post


Link to post
Share on other sites

"IF send() returns me the right number (the total size of the packet), then the packet WILL arrive at destination (unless catastrophic events of course) and it will arrive before anything other that I send after this send, is that right?" If send returns the right number, the data is sent. It will either eventually arrive, or the connection will get dropped.

"After successfully sending the critical message, the server calls shutdown" - You might want to look up the setSockOpt function and the SO_LINGER value.

Alternatively - and probably preferable - wait for the client to confirm that it has connected to the new server before kicking it off the old one. That way you can be sure that the information has got through.

Share this post


Link to post
Share on other sites

"IF send() returns me the right number (the total size of the packet), then the packet WILL arrive at destination (unless catastrophic events of course) and it will arrive before anything other that I send after this send, is that right?" If send returns the right number, the data is sent. It will either eventually arrive, or the connection will get dropped.

"After successfully sending the critical message, the server calls shutdown" - You might want to look up the setSockOpt function and the SO_LINGER value.

Alternatively - and probably preferable - wait for the client to confirm that it has connected to the new server before kicking it off the old one. That way you can be sure that the information has got through.

I found this digging  the web...

https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable

It's true? should I wait for recv to return 0 instead of setting SO_LINGER?

 

The final process will then be:

-server calls shutdown on writing side

-So, the client reads until the recv call return 0 (that will happen only after the server calls shutdown right?)

-then, it just closesocket.

-the server's call to recv will now return 0, and it can now safetely call closesocket.

Share this post


Link to post
Share on other sites

"should I wait for recv to return 0 instead of setting SO_LINGER?" - No. Keep reading:

"Using the shutdown() technique above really only tells us that the remote closed the connection. It does not actually guarantee that all data was received correctly by program B. [..] The best advice is [..]  to have the remote program actively acknowledge that all data was received."

That's why I said you should probably wait for the client to confirm that it has connected to the new server.

Share this post


Link to post
Share on other sites
I take a different approach to closing. Once I know that I've received all the data I need, and I don't have anything I need to send to the server anymore, I just close() the connection on the client side.
On the server side, I will time out a client if I haven't received any data for X seconds, but I will also time out the client if i get an error trying to send() or recv() to/from the client (this means the kernel got the message that my client has closed the connection.)
Lingering/waiting just doesn't buy you anything in these cases. When the server tells you "you need to be talking to this other server," you already know that no more commands to the original server matter, and you shouldn't need any more data from that original server, either.

Separately, how do you detect that you "don't get the data" ?
Specifically, TCP will mush packets together, and also break them apart, so that a single call to send() on a server may end up being received as one, two, three, or more calls to recv() on the client, depending on network conditions.
Thus, if you assume that a single call to send() on the server will correspond to a single call to recv() on the client, you will lose data, and see data that's in the middle of a packet, and such.
Thus, every packet on top of a TCP connection must be preceded by a fixed-length "length" field. For games, you'll seldom want more than a few kilobytes per packet, so a two-byte 16-bit length field should be plenty.
Then, the receiving end can receive until they have at least two bytes (note: you may receive only one in the first call to recv()!) and once it has two or more bytes, check whether it has enough bytes to decode a full packet. If so, decode and remove the packet from the buffer, else keep reading until you have a full packet in the buffer.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this