some advice needed regarding TCP/IP
Author
First off: the definition o "large amount" is "> 50kB" this case.
Here''s what I''m doing and that''s what happens: I set up a connection, create a normal priority thread to send larger chunks of data that I format in the following way:
1) send chunk_start msg (the size of the followig chunk and some other stuff)
2) loop through the chunk and send as many packets as needed
3) send chunk_end msg (this way the other end should make sure it received all of the packets)
The problem that I''m experiencing, is that I have no idea in advance when I have to send the larger chunks - it usually happens sometime during the program execution and several single packets may have been sent just before it. However, evidently the packets are going along very different routes and the single ones that were sent before even the chunk_start message was, arrive among those of the chunk''s, causing the large chunk to become a jumbled mess. Another problem is that chunk_end arrives long before the last pieces of the large chunk itself, causing the chunk to be truncated.
I''m figuring using that kind of a messaging system is wrong... I can''t think of any good method other than labelling all of the packets I''m sending because asking TCP/IP to send a 100+ kB data chunk causes only parts of it to deliver properly. I understand how the system works, but I have no experience in this region, so any pointers would be extremely welcome.
Another question is about the packet size: how can I determine what the current packet payload size used by TCP is and what are the maximum/minimum values it can be set to?
Cheers
You keep mentioning packets in the same breath with TCP/IP which doesn''t jive. TCP/IP is responsible for transmitting a stream of data over a network connection, making sure that the sender and receiver both have exactly the same information. Yes, the data is split into frames when it is eventually sent but that is irrelevant to an application programmer using TCP/IP. If it is relevant, then you probably shouldn''t be using TCP/IP.
If you have a bunch of variable-sized data that you need to transmit using TCP/IP then:
1. Complete any other transfers ahead of it. Since its a stream, you won''t be able to interrupt the transfer of previous data. This isn''t quite true as you could implement an interrupt sentinel or send data out-of-Band but let''s just forget that for now.
2. Send a header describing your data; length, type, etc...
3. Create a buffer for the data on both the sender and receiver side. Since it isn''t likely to complete in one call to send()/recv(), you will need to buffer the data until it is complete.
4. Do a looping send()/recv() until all the data is processed. Done.
As you can see, there normally is no need for an END chunk marker since you already know the exact size to expect via the header.
From what you say in your post, it almost sounds like you are using UDP/IP to send.
If you have a bunch of variable-sized data that you need to transmit using TCP/IP then:
1. Complete any other transfers ahead of it. Since its a stream, you won''t be able to interrupt the transfer of previous data. This isn''t quite true as you could implement an interrupt sentinel or send data out-of-Band but let''s just forget that for now.
2. Send a header describing your data; length, type, etc...
3. Create a buffer for the data on both the sender and receiver side. Since it isn''t likely to complete in one call to send()/recv(), you will need to buffer the data until it is complete.
4. Do a looping send()/recv() until all the data is processed. Done.
As you can see, there normally is no need for an END chunk marker since you already know the exact size to expect via the header.
From what you say in your post, it almost sounds like you are using UDP/IP to send.
> how can I determine what the current packet payload size
> used by TCP is and what are the maximum/minimum values it
> can be set to?
TCP is a byte stream, so there is no ''packet'' per se you need to control at the lower layer levels, and there is no need to determine some optimal buffer size for receiving data in most cases. The default internal TCP buffers is around 20Kb depending on the OS, but that can be changed dynamically with ''setsockopt(...SO_RCVBUF...)''.
> {...} create a normal priority thread
You seem not to control your threads correctly, thus the mess you get at reception. You should thread-lock access to the socket so that only one thread at a time can send its data, including large buffers. If you need to intermix split packets for some reason (you share the link amongst multiple threads), then I suggest you have a header that indicates what kind of packet you are sending, its length and some offset for larger payloads and limit the size to 4Kb (it''s easier on the OS). At reception, you will have to piece together the data chunks from the incoming data stream, according to the header information.
-cb
> used by TCP is and what are the maximum/minimum values it
> can be set to?
TCP is a byte stream, so there is no ''packet'' per se you need to control at the lower layer levels, and there is no need to determine some optimal buffer size for receiving data in most cases. The default internal TCP buffers is around 20Kb depending on the OS, but that can be changed dynamically with ''setsockopt(...SO_RCVBUF...)''.
> {...} create a normal priority thread
You seem not to control your threads correctly, thus the mess you get at reception. You should thread-lock access to the socket so that only one thread at a time can send its data, including large buffers. If you need to intermix split packets for some reason (you share the link amongst multiple threads), then I suggest you have a header that indicates what kind of packet you are sending, its length and some offset for larger payloads and limit the size to 4Kb (it''s easier on the OS). At reception, you will have to piece together the data chunks from the incoming data stream, according to the header information.
-cb
TCP/IP is a streaming protocol so treat it like one. In order to get the best performace you need to think of it as a stream of bytes and double buffer on both ends.
www.icarusindie.com/devzone in the tutorials section has a plug and play winsock class (complete with the standard tutorial bug or two) which demonstrates how to double buffer.
"I can''t think of any good method other than labelling all of the packets"
How can you NOT label the packets and expect it to work? It''s called a header. Use it.
Ben
[ IcarusIndie.com | recycledrussianbrides.com ]
www.icarusindie.com/devzone in the tutorials section has a plug and play winsock class (complete with the standard tutorial bug or two) which demonstrates how to double buffer.
"I can''t think of any good method other than labelling all of the packets"
How can you NOT label the packets and expect it to work? It''s called a header. Use it.
Ben
[ IcarusIndie.com | recycledrussianbrides.com ]
Author
Thanks for the replies, guys - I think I got it now (took me a while to write and test the code). There''s one thing I don''t understand, however - why will the following code cause TCP/IP to hang (it only delivers app. 130 x 255 bytes (PayloadSize = 255, 130 = nr. of calls (out of the required ~500 in this case) to Send()), then chokes on something)
Uncommenting the Sleep() call will fix the problem, though... Does this mean I''m overtrafficking the stream - how can I tell when I should pass more data down the pipe (by polling for the FD_WRITE message?).
Cheers
for(i = 0; i < pLen - PayloadSize; i += PayloadSize) { Send(pData + i, PayloadSize);// Sleep(1); }Uncommenting the Sleep() call will fix the problem, though... Does this mean I''m overtrafficking the stream - how can I tell when I should pass more data down the pipe (by polling for the FD_WRITE message?).
Cheers
Just cause you call Send(...); doesnt mean it will get sent. Yes one of the problems could be filling up the buffer.
You should check the return value from Send and make sure it matches what you expected it to send. If you''re sending data in a tight loop like that chances are youre gonna see the first ones go through just fine, then either a partial send or no send at all until the data is actually sent and the buffer freed up some.
-=[ Megahertz ]=-
You should check the return value from Send and make sure it matches what you expected it to send. If you''re sending data in a tight loop like that chances are youre gonna see the first ones go through just fine, then either a partial send or no send at all until the data is actually sent and the buffer freed up some.
-=[ Megahertz ]=-
Author
Aha - thanks, that''s what I had in mind. Here''s a "flow chart" of the system I worked out, but I can''t be sure if it''ll work ok in real action:
run a parallel thread
if program requests to send data
check if there''s enough room in the TCP/IP buffer space and no pending data currently queued
if there is room, send the data right away
otherwise queue the buffer
in the running thread:
constantly check if there are any queued buffers
if there are, send them in the order they came in
The questions I have are: what priority should I give the thread to be sure all data will be sent as soon as it is queued (in a 3D application, the CPU is generally swamped and intense scenes can cause higer priority threads to suck the lower priority threads dry, giving it too high priority could kill the performance)? Should I suspend the thread when no data is queued or keep it running regardless of the intensity of traffic? Should I add some CRC/checksum to the individual messages or save up on CPU time (in a game)?
run a parallel thread
if program requests to send data
check if there''s enough room in the TCP/IP buffer space and no pending data currently queued
if there is room, send the data right away
otherwise queue the buffer
in the running thread:
constantly check if there are any queued buffers
if there are, send them in the order they came in
The questions I have are: what priority should I give the thread to be sure all data will be sent as soon as it is queued (in a 3D application, the CPU is generally swamped and intense scenes can cause higer priority threads to suck the lower priority threads dry, giving it too high priority could kill the performance)? Should I suspend the thread when no data is queued or keep it running regardless of the intensity of traffic? Should I add some CRC/checksum to the individual messages or save up on CPU time (in a game)?
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
