Archived

This topic is now archived and is closed to further replies.

kuphryn

Determining Data Size For Network Send/Rec :: Winsock

Recommended Posts

Hi. I am studying networking programming from Network Programming For Microsoft Windows, Second Edition by Anthony Jones and Jim Ohlund. I am just about ready to start on a new product that will be my first real Winsock program. I want to design and implement a simple message program (chat) that work over the IP protocol and TCP protocol. I have the basic program core design in my mind. However, I am stuck with one problem and it is a problem that even Jones and Ohlund implied could be difficult to overcome for inexperience network programmers. How do you predetermine the size of the data you receive? Winsock has two basic functions for receiving data: revc (IPv4 or Winsock 1.1+) and WSARecv (IPv6 or Winsock 2). WSARecv is a powerful tool for *predetermined* data size. You can send multiple packages at one time, but the data size must be predetermined (i.e. 10k, 20k, and finally 30k). I will probably use revc since it is more flexible. Even with recv, however, you must give a specific size (byte) to read. Jones and Ohlund recommend sends the size of the data in the first four byte of the data stream. However, what if the user wants to send something that is larger than what a bour byte char variable can hold? Please post any possible solutions. Thanks, Kuphryn

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
If they want to send something larger it is going to take you a very long time to receive, as the max valueof a four byte (32-bit) unsigned integer is, i believe, approximatley 4 billion. It''s not often that people want to send a chat message that is 4 gigabytes in size
The idea is that when you are sending, you get the size of the data you want to send ( using sizeof() ) and send that to the stream first, then you send the actual data.
On the receiving end, the first thing you do is read the first 4 bytes off the stream and convert it to an unsigned int to get the size of the incoming data. You can then use that number to new/malloc a buffer to be sent to the recv() function to get the actual data. If the first call to recv() doesn''t fill the buffer (check the return value of recv() ) then you call it again at the appropriate offset in the buffer where the last input of data ended, and so on until you fill the buffer.

Share this post


Link to post
Share on other sites
Thanks.

Can you give me an example?

Let say I want to sent this message:

"Please give an example of how you would receive this message."

// send(socket, &message, sizeof(message), 0);

Kuphryn

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Ok, this is without error checking, and uses blocking sockets.

unsigned totalSize, currentSize; //totalSize = total size of data
// currentSize = data read off the stream so far

// get the size of the next message
recv( socket, &totalSize, sizeof(unsigned), 0 );

char* buffer = new char[totalSize]; // assumes char''s are single byte

// read data from the socket until we have read totalSize bytes
while ( currentSize += recv( socket, buffer+currentSize, totalSize-currentSize, 0 ) < totalSize );

// then however you want to display the message
displayMessage( buffer );
// free the memory
delete buffer;


That should do the trick, you''d want to change it for non-blocking, and also add plenty of error checking, but thats the basic idea.

Share this post


Link to post
Share on other sites
quote:

Winsock has two basic functions for receiving data: revc (IPv4 or Winsock 1.1+) and WSARecv (IPv6 or Winsock 2). WSARecv is a powerful tool for *predetermined* data size. You can send multiple packages at one time, but the data size must be predetermined (i.e. 10k, 20k, and finally 30k). I will probably use revc since it is more flexible. Even with recv, however, you must give a specific size (byte) to read. Jones and Ohlund recommend sends the size of the data in the first four byte of the data stream. However, what if the user wants to send something that is larger than what a bour byte char variable can hold?



You are incorrect in your statement that recv() is more flexible that WSARecv(). WSARecv is actually a lot more flexible because of its myriad of features. Furthermore, WSARecv can receive as little or as much data as you choose. If you want only four bytes, make sure you specify a WSABUF that has a length of 4. For example:


      
char* databuffer = new char[200];
WSABUF buffer_hdr,
buffer_data;

buffer_hdr.len = 4;
buffer_hdr.buf = databuffer;

buffer_data.len = 196;
buffer_data.buf = &databuffer[4]; // start 4 bytes from start of buffer


WSARecv(s, &buffer_hdr, 1, bytesReceived, 0, 0, 0);
// determine length of data to read

DWORD bytesToRead = (DWORD)buffer_hdr.buf;

// read in the rest of the data

WSARecv(s, &buffer_data, 1, bytesReceived, 0, 0, 0);

// the data appears after the header 'databuffer'

// databuffer[0...3] = length (4 bytes)

// databuffer[4...n] = actual data


// obviously check to make sure WSARecv returned the number of bytes you expected.



Hopefully, this clears things up a bit.

Dire Wolf
www.digitalfiends.com

[edited by - Dire.Wolf on May 7, 2002 9:50:28 AM]

[edited by - Dire.Wolf on May 7, 2002 9:50:53 AM]

Share this post


Link to post
Share on other sites
Yes, your code helped greatly. Thanks!

First, the technique you used is modern because you use WSARevc and the WSABuffer data structure.

There is one drawback (maybe) that I found. For example, you allocate a 200 byte character array. The first four byte will be used as a size indicator. Okay. That work fine. However, what if the size of the actual data after the first four byte is more than 196 bytes? buffer_data will not have room for anything beyond 196 bytes. Unless you are intending on storing all possible data and then coming back for more.

Niklas Lindquist posted a different technique using the revc() function. Here is a link.

http://www.codeproject.com/script/comments/forums.asp?forumid=1647&select=173883&msg=173883#xx173653xx

Thanks again,
Kuphryn

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Find out the length of your message first THEN declare the
buffer_data.len = lengthOfMessage;
or soemthing, understand?

Share this post


Link to post
Share on other sites
Yes but you still need to allocate the memory. In performance critical applications you should avoid continually allocating and deallocating memory, especially for server applications. Continually allocation/deallocation leads to excessive performance hits and memory fragmentation.

I only used a buffer of 200 bytes as an example. In my programs, I normally create a relatively large buffer, say 10000 bytes, and use "message pointers" that point into the buffer. This way I don''t have to continually allocate and deallocate memory. The size of your fixed buffers should be externally configurable so if requirements change (large messages come in) you can change the size of the buffer to suit it.

Dire Wolf
www.digitalfiends.com

Share this post


Link to post
Share on other sites
Thanks.

I have one confusion about byte and character array size that has been bothering me for so long. In your code, Dire.Wolf, you allocated this:

// char MyData[200];

Now, to my experience, MyData can hold up to 200 characters such as "abcdefg......" Okay. However, I saw responses where byte, say 4 byte, can hold up to 2^32 characters.

What is going on? I am confused as to how much data a byte can hold. Does a byte constitute one character or does it constitute 2^8 characters?

Kuphryn

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
A byte is the smallest addressable piece of memory. On most platforms a byte is 8 bits and a char is the same size as a byte.
An integer is 4 bytes (at least on windows and i think linux) and a 4 byte unsigned integer can hold a value (number) up to just over 4 billion.
So an integer doesn hold up to 2^32 characters, it holds a number up to 2^32 and that will be the number of characters following in the stream.
So my program would proceed like this:

Read an integer (4 bytes) from the stream
(Lets say the integer is 23)]
Read 23 char''s (bytes) from the stream. this is the chat message

So the integer you send at the start doesn''t hold any data, it tells you how much data to read off the stream to get the chat message.

Share this post


Link to post
Share on other sites
Ahh.

I understanding now. There is a distinction between what a *variable* can hold and what actual data size relative to characters.

A member recommended using 8 byte to check for the data size instead of 4 byte. Is there a long long data type holding 8 byte integer?

Kuphryn

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Kindof, it depends on your compiler. I think gcc might have a 64-bit "long long" and i think MS VC++ uses __int64. I''m not 100% sure on these though, but check your compiler docs etc.
I really doubt you''d need to use more than a normal unsigned int though, as (particularly for a chat program) it is highly doubtful that you will, or will want to send over 4 gigabytes in a single message...

Share this post


Link to post
Share on other sites
Gosh, I never thought of that application, I''m going to start using 64bit size holes (And I think you can fit 16 Exobytes into one of those, maybe the even the next size up)

It''s not entirely inconceivable that you would never want to send a message larger than 64k - suppose you want to do something like serialize and send an IMediaSample over the pipe for a digital CCTV application, so using 32bit isn''t a terrible idea. However, in most applications, you probably won''t send messages larger than 64k, so you can use a 16bit hole to store the size.

Share this post


Link to post
Share on other sites
Honestly, I''d stick the limit at 255 characters (1 byte/8 bits). Anyone who types more than that many characters in a chat message needs to be shot anyway.

BTW, this doesn''t include message boards, where I''ve commonly been guilty of making posts in the 12-16000 character range :D

Share this post


Link to post
Share on other sites
quote:
(And I think you can fit 16 Exobytes into one of those, maybe the even the next size up)


Not sure if it''s Exobytes, but it''s 18 sextillion bytes, or maybe it''s pentillion, but I thought that they skipped pentillion and went from quadrillion to sextillion.

OK, just checked, it''s quintillion.

Share this post


Link to post
Share on other sites