Jump to content

  • Log In with Google      Sign In   
  • Create Account


How do i determine send/recv buffer size


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
14 replies to this topic

#1 Shawn619   Members   -  Reputation: 342

Like
0Likes
Like

Posted 10 January 2014 - 02:24 PM

In terms of Winsock,

 

if the client sometimes needs to send 100 bytes and sometimes 1000 bytes for example, how is the server supposed to know before-hand the buffer size for the incoming send() from the client of 100 or 1000 bytes that i would need in recv() on the server?

 

Should i just set them both to a limit that i know for certain it won't reach, say 2048:

Client:
send(socket, cstr, 2048);
 
Server:
recv(socket, recvbuf, 2048);

Or should i do like the Winsock example and save packet space by  when i send() by capping the buffer size at the c-string's size:

Client:
send(socket, cstr, strlen(cstr));
 
Server:
recv(socket, recvbuf, 2048);

Questions:

How do i determine the buffer size to use?

Should the send()'s buffer size always be capped at the length of data im sending(strlen(ctr)), or should it just be 2048 for example across the whole program?

Am i wasteing 2000+ bytes worth of packets if i do something like this

send(socket, "Username=Jimmy".c_str(), 2048);//not proper syntax i know

 ?

 



Sponsor:

#2 Nypyren   Crossbones+   -  Reputation: 3728

Like
3Likes
Like

Posted 10 January 2014 - 03:18 PM

You can deal with this in a few ways.  The two most obvious methods are:

  • Start each message with a number that indicates how many more bytes you're sending (usually you would send this in binary in network byte order but you could use text as well - you'd just need to terminate the number's text in that case.  HTTP sends content-length as plain text for example).
  • End each message with a special terminator sequence (like how C strings have null terminators).

I usually prefer the first method because it's easier to write a receive loop if you know the expected number of bytes ahead of time.

 

In the first method, your receive code will look somewhat like this:

  • 'select' to find out if there's any data available to read.  (Optional - if your code has nothing else to do you can skip this step and go straight to reading)
  • 'read' the number to find out the size of the next message (if you use binary you can use a fixed number of bytes such as 2 or 4.  If you use text, you have to read until you reach the terminator character).
  • 'read' in a loop until you receive that many bytes.

Edited by Nypyren, 10 January 2014 - 03:23 PM.


#3 ByteTroll   Prime Members   -  Reputation: 1170

Like
0Likes
Like

Posted 10 January 2014 - 05:49 PM

Shawn, the last project I worked on used Nypyren's first suggestion and it is by far the eaiest way to manage this problem.  Just send the size first.


▬▬▬▬▬▬▬▬▬▬▬ஜ۩۞۩ஜ▬▬▬▬▬▬▬▬▬▬▬▬▬
I see the future in 1's and 0's
▬▬▬▬▬▬▬▬▬▬▬ஜ۩۞۩ஜ▬▬▬▬▬▬▬▬▬▬▬▬▬

"This is called programming. The art of typing shit into an editor/IDE is not programming, it's basically data entry. The part that makes a programmer a programmer is their problem solving skills." - Serapth

#4 TGrey   Members   -  Reputation: 132

Like
0Likes
Like

Posted 11 January 2014 - 02:42 AM

My server did it in this way, packet:

byte - packet id

signed short (2 bytes) - packet size

body (N bytes) - packet body

 

First alloc any size you want, put there required information, when you done, set packet id, calculate size of output buffer and set it to 2,3d bytes.

 

If your packet has constant size, and has id = 1, for example:

id(byte)

size(sshort)

account(char 16)

password(char 16)

save_password(byte(bool))

total 36 bytes, there are always 36 bytes, and you know that client have to read 36 bytes for packet with id 1.

 

If you have dynamic size packet with id 2:

id(byte)

size(sshort)

message(string)

You have always 3 bytes in begin and some bytes in the end. You make dynamic buffer, put there your string, for example("Hello, there!") - 13 bytes. And set it to packet.

id: 2

size: 16

message: "Hello, there!"

You read first 3 bytes to get id, and size, and then read whole string.

 

But i guess dynamic strings you can set only to last value. If you have string in middle of the packet, you should know about that and set constant size for this string.


Edited by TGrey, 11 January 2014 - 05:54 AM.


#5 fholm   Members   -  Reputation: 262

Like
0Likes
Like

Posted 11 January 2014 - 05:29 AM

The way I have solved this is the following:

 

Set both the send and recv buffers to some pre-defined max-size, for me that is 2048 bytes or 2kb. No packet sent can be larger than this, ever. This allows you to pre-allocate as many 2kb buffers are you need to handle all incoming packets, and re-cycle them by just zeroing the memory again, without the need to free/alloc new memory every time. My game is also written in C#, so this lessens the pressure on the GC a lot, allowing me to recycle packet buffers and not having to re-size them, ever.

 

So, about knowing how much "data" you can read from each received packet, most (all?) socket layers return the amount of bytes read from the recvfrom call. But since we usually deal with bits and not bytes, this only gives us an upper bound on the amount of bits (received bytes * 8) that we have. There are three different ways to deal with getting the exact data out of the packet that I have found, as an example I am going to say that I need to send 5 RPC calls and 2 Player Data objects

  1. The most common and obvious one is to have the first two bytes of the packet be the exact size, in bits, of the packet, so the packet would look like this: [size][rpc][rpc][rpc][rpc][rpc][playerData][playerData]

  2. Know what data you sent in the packet, and pre-fix each block of data with the count of that type of data, for example if i need to send five RPC calls and eight player data objects, my packet would look like this: [5][rpc][rpc][rpc][rpc][rpc][2][playerData][playerData]

  3. Before each segment of data, write one bit set to 1. Before you read each piece of data, you read this bit, if it's 1 you should read the data, if it's 0 you are done, so the packet would look like this: [1][rpc][1][rpc][1][rpc][1][rpc][1][rpc][1][playerData][1][playerData]

Personally i use the third approach because the other two methods require you to "go back" and write the size/count after you have serialized all the data, and it creates an awkward logic flow. The third approach allows each write to be completely self-contained with no need to refer to anything outside of itself.



#6 hplus0603   Moderators   -  Reputation: 4978

Like
1Likes
Like

Posted 11 January 2014 - 11:35 AM

The question is concerning to me, because it seems like you think that a single call to send() will end up in a single call to recv() on the other end.

This is not the case.

 

If the sender sends one message of 15 bytes, one message of 238 bytes, and one message of 1 byte, the recipient may receive:

- one message of 15 bytes, and one message of 239 bytes

- or one message of 2 bytes, one message of 4 bytes, one message of 8 bytes, one message of 16 bytes, one message of 32 bytes, one message of 64 bytes, and one message of 128 bytes

- or one message of 254 bytes

- or some other sequence

This assumes there is no failure in the network, in which case the recipient may receive nothing.

 

The receiver in a TCP based system should pretty much always keep a fixed-size receive buffer, and each time calling recv(), call recv() for as much as is left in that buffer. If you use a cyclic buffer, it means you don't have to copy/move data as you dequeue it, but you have to deal with the "break" at the end.

 

Once you have received data into the buffer, you can start looking at it to see if you have received a full message from the other end (whatever that means,) and if so, process and remove it. Repeat.


enum Bool { True, False, FileNotFound };

#7 Shawn619   Members   -  Reputation: 342

Like
0Likes
Like

Posted 11 January 2014 - 06:00 PM

The question is concerning to me, because it seems like you think that a single call to send() will end up in a single call to recv() on the other end.

This is not the case.

 

If the sender sends one message of 15 bytes, one message of 238 bytes, and one message of 1 byte, the recipient may receive:

- one message of 15 bytes, and one message of 239 bytes

- or one message of 2 bytes, one message of 4 bytes, one message of 8 bytes, one message of 16 bytes, one message of 32 bytes, one message of 64 bytes, and one message of 128 bytes

- or one message of 254 bytes

- or some other sequence

This assumes there is no failure in the network, in which case the recipient may receive nothing.

 

I don't know if you're referring to Nagle's algorithm, but i've never had a situation where a remote client would recv() partial bytes, like 2 bytes of 100 bytes for example. Either the client receives the server's send() message in full or a winsock error message (ie: error code 10035) is created, per the return value of recv(). Or maybe you're referring to a client that has a network connection that's cutting in and out 

 

 

The receiver in a TCP based system should pretty much always keep a fixed-size receive buffer, and each time calling recv(), call recv() for as much as is left in that buffer. If you use a cyclic buffer, it means you don't have to copy/move data as you dequeue it, but you have to deal with the "break" at the end.

 

Once you have received data into the buffer, you can start looking at it to see if you have received a full message from the other end (whatever that means,) and if so, process and remove it. Repeat.

 

Good, this is exactly what i'm doing and it's working perfect.  



#8 fholm   Members   -  Reputation: 262

Like
1Likes
Like

Posted 12 January 2014 - 01:40 AM

I don't know if you're referring to Nagle's algorithm, but i've never had a situation where a remote client would recv() partial bytes, like 2 bytes of 100 bytes for example. Either the client receives the server's send() message in full or a winsock error message (ie: error code 10035) is created, per the return value of recv(). Or maybe you're referring to a client that has a network connection that's cutting in and out 

 

 

Maybe it's rare, but it does happen and it is a case that needs to be handled.



#9 ApochPiQ   Moderators   -  Reputation: 14299

Like
1Likes
Like

Posted 12 January 2014 - 04:10 AM

I don't know if you're referring to Nagle's algorithm, but i've never had a situation where a remote client would recv() partial bytes, like 2 bytes of 100 bytes for example. Either the client receives the server's send() message in full or a winsock error message (ie: error code 10035) is created, per the return value of recv(). Or maybe you're referring to a client that has a network connection that's cutting in and out


It's easy to have this happen if you encounter packet fragmentation, e.g. sending a packet larger than routable MTU. It isn't exactly rare if you're sending nontrivial amounts of data.

#10 hplus0603   Moderators   -  Reputation: 4978

Like
0Likes
Like

Posted 12 January 2014 - 03:12 PM

 

 

 

 

My guess is you simply haven't been paying attention. This is not particularly rare -- over the greater internet, with your own hosting and a remote user, you will possibly see it on up to a percent of all packets.

I am not referring to the bad case of networks that are physically unreliable. A simple packet collision on any segment between sender and receiver is enough to make this happen.


enum Bool { True, False, FileNotFound };

#11 Shawn619   Members   -  Reputation: 342

Like
0Likes
Like

Posted 13 January 2014 - 04:35 AM

It's easy to have this happen if you encounter packet fragmentation, e.g. sending a packet larger than routable MTU. It isn't exactly rare if you're sending nontrivial amounts of data.

 

Oh ok i see, i'll account for this, thanks.

 

 

My guess is you simply haven't been paying attention. 

Relax big boy, there's no reason for personal attacks, you should know that.


Edited by Shawn619, 13 January 2014 - 04:36 AM.


#12 hplus0603   Moderators   -  Reputation: 4978

Like
0Likes
Like

Posted 13 January 2014 - 01:22 PM

I don't think "I guess you haven't been paying attention" is an attack in this context; my apologies if you read it as such. I merely meant that you don't see these things unless you explicitly look for them, but once you do, you will see them with some regularity. If you're not explicitly looking for merged packets, you will "only" see "something weird" which you'll have no way to investigate (if you even pay attention at the time it happens.)

 

How about "I guess you haven't been explicitly looking"?


enum Bool { True, False, FileNotFound };

#13 Shawn619   Members   -  Reputation: 342

Like
0Likes
Like

Posted 13 January 2014 - 02:46 PM

I don't think "I guess you haven't been paying attention" is an attack in this context; my apologies if you read it as such. I merely meant that you don't see these things unless you explicitly look for them, but once you do, you will see them with some regularity. If you're not explicitly looking for merged packets, you will "only" see "something weird" which you'll have no way to investigate (if you even pay attention at the time it happens.)

 

How about "I guess you haven't been explicitly looking"?

That sounds much better. You may want to be cautious with your phrasing, because "My guess is you simply haven't been paying attention." infers the other person was too lazy to notice something.

 

I accept your apology. I don't enjoy calling random people out in a gaming forum.



#14 ApochPiQ   Moderators   -  Reputation: 14299

Like
0Likes
Like

Posted 13 January 2014 - 03:00 PM

Dude, you need to grow a thicker skin.

 

Especially on the internet it's a bad idea to assume people are malicious with their wording. All we have is text, and human communication is far more nuanced than just the verbal layer.

 

 

Nobody attacked you, nobody insulted you, and you have nobody to "call out."



#15 Shawn619   Members   -  Reputation: 342

Like
0Likes
Like

Posted 14 January 2014 - 01:37 AM

Dude, you need to grow a thicker skin.

 

Especially on the internet it's a bad idea to assume people are malicious with their wording. All we have is text, and human communication is far more nuanced than just the verbal layer.

 

 

Nobody attacked you, nobody insulted you, and you have nobody to "call out."

Let's not gang up on me now, I already accepted his apology,






Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS