Receive Buffer Size
Author
If, when receiveing, you call recv() and set the buffer size to, say 5. What if there are 10 bytes of data waiting? What happens?
-Greg
For connection-oriented sockets (type SOCK_STREAM for example), calling recv will return as much information as is currently available—up to the size of the buffer supplied.
For UDP. If the datagram or message is larger than the buffer supplied, the buffer is filled with the first part of the datagram, and recv generates the error WSAEMSGSIZE. For unreliable protocols (for example, UDP) the excess data is lost.
--
Martin Piper
Network programming made easy with ReplicaNet
http://www.ReplicaNet.com/
For UDP. If the datagram or message is larger than the buffer supplied, the buffer is filled with the first part of the datagram, and recv generates the error WSAEMSGSIZE. For unreliable protocols (for example, UDP) the excess data is lost.
--
Martin Piper
Network programming made easy with ReplicaNet
http://www.ReplicaNet.com/
Author
But for connections like TCP the data is still in the buffer? Would it be bad practice to just read a part of the data if I knew that there was probably more in the buffer? I was thinking of just reading in the header which has the size of the rest of the packet and then calling recv again with the max buffer size as the size of my packet. I would do this so I would not have two packets at once. Is this a bad idea?
-Greg
-Greg
Use a header and move data pulled from recv into a buffer. Using the header information you then break down the buffer into complete messages. Don''t worry about how much recv returns. It''s irrelavent.
My winsock code (available at my site) is fully buffered. The main program simply calls a method of the winsock class which returns either a complete message or message type -1 indicating no message needs to be processed.
You also want to buffer messages you intend to send. I empty the send buffer (actually send the messages out) once per iteration.
This way N messages are treated like one which is a lot more efficient.
Ben
IcarusIndie.com
KalvinB - (to Jessika) do you accept Jesus as your lord and savior
Jessika - Sure I can accept all forms of payment.
My winsock code (available at my site) is fully buffered. The main program simply calls a method of the winsock class which returns either a complete message or message type -1 indicating no message needs to be processed.
You also want to buffer messages you intend to send. I empty the send buffer (actually send the messages out) once per iteration.
This way N messages are treated like one which is a lot more efficient.
Ben
IcarusIndie.com
KalvinB - (to Jessika) do you accept Jesus as your lord and savior
Jessika - Sure I can accept all forms of payment.
If you''re using TCP, then it makes a lot of sense to do packets like:
and read it somewhat like so:
Some people will say that entering the network stack twice to receive one packet is inefficient, but compared to the slowness of any available consumer networking, the overhead of this is zippo, zilch and nada. Heck, even the malloc() probably costs nothing when amortized to the enormous sloth of even a DSL connection.
Another question though: why are you using TCP? TCP is bad if you''re sending data that needs to get there "ASAP". I e, the position/direction/action packets of a FPS need to be UDP so that a single dropped packet doesn''t cause indigestion, and then you need to figure out how to recover from lost packets. TCP is not a good fit there.
struct packet { short type; short size;}; and read it somewhat like so:
struct packet p;recv( input, &p, sizeof( p ) );p.type = ntohs( p.type );p.size = ntohs( p.size );char * data = p.size ? malloc( size ) : 0;if( ps.size ) { recv( input, data, p.size );}onPacketReceived( p, data, size );free( data ); Some people will say that entering the network stack twice to receive one packet is inefficient, but compared to the slowness of any available consumer networking, the overhead of this is zippo, zilch and nada. Heck, even the malloc() probably costs nothing when amortized to the enormous sloth of even a DSL connection.
Another question though: why are you using TCP? TCP is bad if you''re sending data that needs to get there "ASAP". I e, the position/direction/action packets of a FPS need to be UDP so that a single dropped packet doesn''t cause indigestion, and then you need to figure out how to recover from lost packets. TCP is not a good fit there.
Author
Thank you anon, that is what I was going to do but was not sure if the overhead of calling recv twice was really high. Why I am using TCP? I don''t really know. So far it seems to be the best one for me. I may switch over to UDP once I find problems with TCP.
That would work anon if TCP didn''t break up messages. Not only is it inefficient and rediculously over complicated but it doesn''t work. It won''t even work on a LAN.
You can not determine what message has been recieved or how large it is right off of recv.
You do it the right way that actually works by storing everything that comes through in recv into a buffer that is created at compile time. Mine is around 64KB which is overkill because messages should be moved out fast enough that you would never have that much information pending.
You must fully buffer because you have no idea if what you recv''d was an incomplete message, a full message, or multiple messages. You can''t know until you have enough data in your buffer to form at least one full header.
My TCP/IP Winsock class which works efficiently and right.
Ben
IcarusIndie.com
KalvinB - (to Jessika) do you accept Jesus as your lord and savior
Jessika - Sure I can accept all forms of payment.
You can not determine what message has been recieved or how large it is right off of recv.
You do it the right way that actually works by storing everything that comes through in recv into a buffer that is created at compile time. Mine is around 64KB which is overkill because messages should be moved out fast enough that you would never have that much information pending.
You must fully buffer because you have no idea if what you recv''d was an incomplete message, a full message, or multiple messages. You can''t know until you have enough data in your buffer to form at least one full header.
My TCP/IP Winsock class which works efficiently and right.
Ben
IcarusIndie.com
KalvinB - (to Jessika) do you accept Jesus as your lord and savior
Jessika - Sure I can accept all forms of payment.
quote:Original post by KalvinB
That would work anon if TCP didn''t break up messages. Not only is it inefficient and rediculously over complicated but it doesn''t work. It won''t even work on a LAN.
You can not determine what message has been recieved or how large it is right off of recv.
This seems strange. TCP is stream-based, so it has no idea what a "message" is; it simply delivers a flow of bytes to the designated socket. As a result, there is no reason why you couldn''t determine what message has been received. The procedure would be just as Anon has suggested: you''d read in some sort of type parameter to identify the incoming data, read in a size parameter, and then use the size parameter to read in the data. Why doesn''t this work?
quote:
You do it the right way that actually works by storing everything that comes through in recv into a buffer that is created at compile time. Mine is around 64KB which is overkill because messages should be moved out fast enough that you would never have that much information pending.
What you''ve described sounds like the beginnings of a method to implement a reliable layer on top of UDP by storing individual packets in a buffer until they could be merged into a continuous message, which could then be processed. (And, btw, I agree with you, 64Kb is a truly huge amount of space for relatively small network messages.)
quote:
You must fully buffer because you have no idea if what you recv''d was an incomplete message, a full message, or multiple messages. You can''t know until you have enough data in your buffer to form at least one full header.
How does that happen? Using the procedure Anon outlined with blocking sockets, you''re guaranteed to begin receiving at the beginning of a message. You don''t stop receiving until you''ve received the data block, which is the end of the message.
________________________________________________
RapscallionGL development continues...
"Optimal decisions, once made, do not need to be changed." - Robert Sedgewick, Algorithms in C
"As a result, there is no reason why you couldn''t determine what message has been received"
Okay then, let''s break out the dictionary:
message: a variable length of bytes whose purpose and length is known by a header
header: a fixed number of bytes used to identify and describe a message
"you''d read in some sort of type parameter to identify the incoming data, read in a size parameter, and then use the size parameter to read in the data. Why doesn''t this work?"
How do you know you got the whole type parameter? How do you know what came in was even the type parameter? What if you''re still waiting for 3 bytes a message and recv returns 5. With anon''s way you have absolutly no way to tell what to do. You just screwed up your next message.
"What you''ve described sounds like the beginnings of a method to implement a reliable layer on top of UDP by storing individual packets in a buffer until they could be merged into a continuous message, which could then be processed."
This will obviously come as a shock but TCP has to be handled the same way. All you''re getting is a stream of bytes. That''s why you just buffer whatever comes in and figure out what it is later. When you have a number of bytes in the buffer greater than or equal to the number of bytes defined by the header, only then can you accuratly determine what message has come in and if the whole message is there or more than the message is there or more message is still comming.
"Using the procedure Anon outlined with blocking sockets"
Why in the world would you use blocking sockets? You do not use blocking sockets.
Ben
IcarusIndie.com
KalvinB - (to Jessika) do you accept Jesus as your lord and savior
Jessika - Sure I can accept all forms of payment.
Okay then, let''s break out the dictionary:
message: a variable length of bytes whose purpose and length is known by a header
header: a fixed number of bytes used to identify and describe a message
"you''d read in some sort of type parameter to identify the incoming data, read in a size parameter, and then use the size parameter to read in the data. Why doesn''t this work?"
How do you know you got the whole type parameter? How do you know what came in was even the type parameter? What if you''re still waiting for 3 bytes a message and recv returns 5. With anon''s way you have absolutly no way to tell what to do. You just screwed up your next message.
"What you''ve described sounds like the beginnings of a method to implement a reliable layer on top of UDP by storing individual packets in a buffer until they could be merged into a continuous message, which could then be processed."
This will obviously come as a shock but TCP has to be handled the same way. All you''re getting is a stream of bytes. That''s why you just buffer whatever comes in and figure out what it is later. When you have a number of bytes in the buffer greater than or equal to the number of bytes defined by the header, only then can you accuratly determine what message has come in and if the whole message is there or more than the message is there or more message is still comming.
"Using the procedure Anon outlined with blocking sockets"
Why in the world would you use blocking sockets? You do not use blocking sockets.
Ben
IcarusIndie.com
KalvinB - (to Jessika) do you accept Jesus as your lord and savior
Jessika - Sure I can accept all forms of payment.
Alright, I''m beginning to understand your method after reading the last post. What you basically do is read *all* the available pending data on a socket into a buffer and process it later. This seems strange to me because you might as well take it in chunks and process as you go.
For example, suppose you have a four byte header followed by a twenty byte message and that your socket has received two of these header/message combinations for a total of forty-eight bytes pending in the network buffer.
Your method reads all forty-eight bytes into the buffer immediately and processes the information later, so you''re forced to deal with that buffer containing multiple messages, incomplete messages, etc.
Anon''s method reads exactly the number of bytes needed for a header, determines the type and size of the data payload, and then reads a number of bytes exactly equal to the payload''s size. Thus, at that point, one complete message has been processed and another twenty-four byte message remains in the network buffer. The process then repeats, and you end up absorbing entire messages at a time; you''re never in the middle of a message or have to face multiple messages during a single processing cycle.
See above; since I only absorb whole messages at a time, I don''t have to worry about remembering some random part of another message and piecing them together.
Or you process the data and automatically delineate it into separate messages as it comes. Both methods apparently work, but it seems easier and less convoluted to deal with entire messages rather than manipulate a buffer full of partial segments of messages. Now with UDP, you obviously have no choice due to the nature of the protocol, but with TCP, why go through the trouble? Overall, no, TCP does not have to be handled in the same way.
Even under your method, that statement is not entirely correct. Your post-recv() processing tells you to expect more of the same message and the exact size of the rest of that message, so if you determine that you''ve got an incomplete message, you can always immediately take x bytes off the front of the newly-received buffer and attribute those to the stored, partial message.
Perhaps a particular solution requires nothing more than blocking sockets. Still, the same process-as-you-go method works with asynchronous sockets, overlapped I/O, completion ports, etc.
________________________________________________
RapscallionGL development continues...
"Optimal decisions, once made, do not need to be changed." - Robert Sedgewick, Algorithms in C
For example, suppose you have a four byte header followed by a twenty byte message and that your socket has received two of these header/message combinations for a total of forty-eight bytes pending in the network buffer.
Your method reads all forty-eight bytes into the buffer immediately and processes the information later, so you''re forced to deal with that buffer containing multiple messages, incomplete messages, etc.
Anon''s method reads exactly the number of bytes needed for a header, determines the type and size of the data payload, and then reads a number of bytes exactly equal to the payload''s size. Thus, at that point, one complete message has been processed and another twenty-four byte message remains in the network buffer. The process then repeats, and you end up absorbing entire messages at a time; you''re never in the middle of a message or have to face multiple messages during a single processing cycle.
quote:Original post by KalvinB
How do you know you got the whole type parameter? How do you know what came in was even the type parameter? What if you''re still waiting for 3 bytes a message and recv returns 5. With anon''s way you have absolutly no way to tell what to do. You just screwed up your next message.
See above; since I only absorb whole messages at a time, I don''t have to worry about remembering some random part of another message and piecing them together.
quote:
This will obviously come as a shock but TCP has to be handled the same way. All you''re getting is a stream of bytes. That''s why you just buffer whatever comes in and figure out what it is later.
Or you process the data and automatically delineate it into separate messages as it comes. Both methods apparently work, but it seems easier and less convoluted to deal with entire messages rather than manipulate a buffer full of partial segments of messages. Now with UDP, you obviously have no choice due to the nature of the protocol, but with TCP, why go through the trouble? Overall, no, TCP does not have to be handled in the same way.
quote:
When you have a number of bytes in the buffer greater than or equal to the number of bytes defined by the header, only then can you accuratly determine what message has come in and if the whole message is there or more than the message is there or more message is still comming.
Even under your method, that statement is not entirely correct. Your post-recv() processing tells you to expect more of the same message and the exact size of the rest of that message, so if you determine that you''ve got an incomplete message, you can always immediately take x bytes off the front of the newly-received buffer and attribute those to the stored, partial message.
quote:
Why in the world would you use blocking sockets? You do not use blocking sockets.
Perhaps a particular solution requires nothing more than blocking sockets. Still, the same process-as-you-go method works with asynchronous sockets, overlapped I/O, completion ports, etc.
________________________________________________
RapscallionGL development continues...
"Optimal decisions, once made, do not need to be changed." - Robert Sedgewick, Algorithms in C
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
