Details on UDP Security (or lack thereof)
Author
I can google/forum search any number of posts and they all say essentially the same thing about UDP: it's faster than TCP, but far less reliable in that it has no packet sorting or security (hence why it's faster). This I understand quite readily, and I've read that UDP is best used for "momentary" updates, like position and motion vectors, whereas TCP is best used for "manditory" updates, like chat messages and the like.
My real question isn't whether or not UDP is secure or not, but really, how secure it is and what the factors are that influence packet loss or shuffling and how to mitigate those factors.
So:
--What is the average rate of packet loss? I doubt this can easily be stated, but the factors that lead towards higher or lower packet loss on a given connection would be greatly appreciated.
--Likewise, what leads to packet shuffling (out of order), and how often does it occur?
--What are methods of overcomming packet loss for important messages while transmitting over UDP? Gamasutra's article on Age of Empires made a good example: important packets get flagged for a reply. If a reply is not received for a flagged packet within a certain amount of time (a guesstimate of how long it would take for the server to receive and reply to a flagged packet), it's sent again and again until it's reception is confirmed. Are there other, superior methods to this, or optimizations/problems I may not have thought of?
My situation is quite ironic. Before I was a staunch TCP advocate, and scoffed at the idea of using an insecure transmission protocol. I wrote annoyingly complex socket classes to handle an unknown number of simultaneous connections, then equally annoying functions to create new connections and store them in the aforementioned class... On a whim, I tried something similar in UDP and had something working and so very simple in seconds that I was sold at that moment, and haven't honestly looked back. Now I'm trying to have the best of both worlds, as near as I can. Speed and perhaps not security, but reliability.
I just recently starting writing a UDP system for my project, abandoning TCP as well.
From what I've read on the internet, apparently UDP is pretty reliable these days with all the new networking hardware that is being used everywhere. But it's still not 100% reliable.
Same goes for packet disordering. Maybe about one or two percent get dropped or disordered, depending on how far it's traveling, and how many routers it hops.
I setup my system to include a special 'ID' byte as the first byte in all outgoing packets. If the packet doesn't need to be sent reliably, then it sets the first byte to 255/0xFF.
On the end of all outgoing packets, I append the last 3 received reliable ID bytes from whoever the outgoing packet is headed for as a way to send acknowledgement of received reliable packets.
These extra bytes are sent on ALL outgoing traffic, reliable or not, but because my game server is handling everything through one socket/port, I have to keep track of separate 3-byte lists for each 'client' that is 'connected' to the game, one list for each client who I could be receiving reliable packets from that need acknowledgement of their reception. 3 bytes per client is no big deal.
I queue all my outgoing packets, and send them out at a set interval, typically the most that ever go out is about 2 or 3, but usually only one is being sent with each of these sendqueue updates.
If the network comes across a 'reliable' packet in the outgoing queue (1st byte isn't 255) it simply sends the packet, but doesn't remove it from the queue.. Unreliable packets (1st byte is 255) are removed from the queue immediately after sending..
The only thing that will remove an outgoing reliable packet from the queue is seeing its ID byte in the end 3 bytes of an incoming packet. But this requires that there are no idle moments, a useful packet has to arrive that the acknowledgement bytes are piggybacking on, otherwise the queue will just keep sending it like there's no tomorrow, which should only occur if the 'connection' is lost, at least on paper anyway..
Quote:Original post by Zouflain
--What are methods of overcomming packet loss for important messages while transmitting over UDP? Gamasutra's article on Age of Empires made a good example: important packets get flagged for a reply. If a reply is not received for a flagged packet within a certain amount of time (a guesstimate of how long it would take for the server to receive and reply to a flagged packet), it's sent again and again until it's reception is confirmed. Are there other, superior methods to this, or optimizations/problems I may not have thought of?
TCP is basically that. A protocol that ensures in-order, reliable delivery relatively efficiently. If you want to write a very efficient reliable stream system on top of UDP, you'll end up re-writing something similar to TCP.
Basically with UDP reliable messaging, start simple (like the algo you described), and work towards optimisations later. If you need lots of reliability, TCP would be easier. If you need only some reliability, where latency doesn't really matter, write something simple as it's likely not to be your bottleneck.
Quote:
My real question isn't whether or not UDP is secure or not, but really, how secure it is and what the factors are that influence packet loss or shuffling and how to mitigate those factors.
As for 'security', if you mean it in the form of encryption, TCP isn't any more secure than UDP. if you mean it as reliability, then yes and no. UDP can receive anything from anyone ,so you need to do some filtering / rejection at your end. However, UDP will never drop a connection, because there aren't any, but as soon as you want something reliable and basically a two way communication (the other side sends acks), then you'll need to worry about point to point connections, just like TCP does.
You work around packet ordering by inserting a sequence number in your packets, then you can decide to ignore a packet with an older sequence number.
You deal with packet loss by either re-transmitting the lost data (similar to a reliable protocol), ignore it all together (if you just send positions, and you miss one transmission, it wont matter that much), or use data-management schemes such as delta-compression.
Note that out of order packets can mean packet loss, since you usually will discard older packets (out of order) right off.
Quote:
My situation is quite ironic. Before I was a staunch TCP advocate, and scoffed at the idea of using an insecure transmission protocol. I wrote annoyingly complex socket classes to handle an unknown number of simultaneous connections, then equally annoying functions to create new connections and store them in the aforementioned class... On a whim, I tried something similar in UDP and had something working and so very simple in seconds that I was sold at that moment, and haven't honestly looked back. Now I'm trying to have the best of both worlds, as near as I can. Speed and perhaps not security, but reliability.
I found working with UDP a bit of a pain. Usually, I work with only one UDP socket, that receives anything from updates, client connection requests, client quit notifications, ect...
In the end, I end up re-writing a handshake mechanism not dissimilar to TCP-IP, and have to manage a connection layer (a sort of sub-socket on top of the UDP socket). Since you can receive anything from anyone on a UDP socket, You need to assign some verification, registration and identification to each clients that try to connect to you, so you can route packets to the right channels. Also, you can setup a NAT punch-through and encryption mechanism as well, which is what XBL (UDP based socket) and other libraries do to secure a point-to-point connection.
Then you need to detect disconnections (implement a timeout mechanism, maybe some recovery system), and so on... What you end up with are virtual connections, which is what you will find with many network libraries, as the basic layer of communication. So, it's not all fun fun fun...
[Edited by - oliii on July 24, 2009 6:30:35 PM]
furthermore, UDP packets are limited in size, usually 64K. The packet segmentation and reconstruction is handled internally. Segmentation can also be a source of packet loss. If a segment gets lost or out of order, the whole packet is rejected.
TCP works differently, so theoretically, you could send megabytes, but your calls to send() will most likely be blocked (is the socket buffer is full and waits to send some data), return a memory overflow error, or the number of bytes successfully written into the send queue.
[Edited by - oliii on July 24, 2009 6:58:17 PM]
TCP works differently, so theoretically, you could send megabytes, but your calls to send() will most likely be blocked (is the socket buffer is full and waits to send some data), return a memory overflow error, or the number of bytes successfully written into the send queue.
[Edited by - oliii on July 24, 2009 6:58:17 PM]
The topic's title is a bit unfortunate, as neither UDP or TCP are "secure", the correct word is "reliable". Also, you can't say that UDP is "faster", it is not. TCP and UDP are exactly the same speed, they go over the same wire. However, TCP may stall more and be less responsive.
The only real difference is that TCP works on a reliable stream whereas UDP works on datagrams (messages, or chunks). For reliability, you must resend packets if they were lost, with either protocol. TCP does it under the hood, for UDP you have to do it by hand.
Now, the reason why UDP may seem "faster" is that individual datagrams don't necessarily arrive in the correct order, and datagram #254353 doesn't know about datagram #254352. The sender keeps yanking them out, and the receiver gets them, or doesn't.
Since TCP works on a continuous stream, it must stall everything until all previous datagrams have been received. If one is missing, it will wait for it. This may introduce extra latencies. Also, TCP has (unless you turn it off) a mechanism to collapse several bits of small data into one packet. UDP doesn't have that, it just sends what you give to it, and you have the guarantee that either the chunk arrives as-is, or not at all (which is a good thing!).
Now, for the actual numbers of lost packets and what influences them, the answer to all these is: "you don't know, and you will never know". You can probably expect no more than 1-2% packet loss for most "normal" situations, but there is no guarantee for that.
Routers may favour one protocol over another, so if someone starts pinging a nearby server and the router is already a bit short on buffers, it might just drop your UDP or TCP traffic in favour of the "more important" ICMP. Your hoster might give TCP a higher weight, and some companies/universities might not forward UDP to/from the internet at all. You just don't know.
The only real difference is that TCP works on a reliable stream whereas UDP works on datagrams (messages, or chunks). For reliability, you must resend packets if they were lost, with either protocol. TCP does it under the hood, for UDP you have to do it by hand.
Now, the reason why UDP may seem "faster" is that individual datagrams don't necessarily arrive in the correct order, and datagram #254353 doesn't know about datagram #254352. The sender keeps yanking them out, and the receiver gets them, or doesn't.
Since TCP works on a continuous stream, it must stall everything until all previous datagrams have been received. If one is missing, it will wait for it. This may introduce extra latencies. Also, TCP has (unless you turn it off) a mechanism to collapse several bits of small data into one packet. UDP doesn't have that, it just sends what you give to it, and you have the guarantee that either the chunk arrives as-is, or not at all (which is a good thing!).
Now, for the actual numbers of lost packets and what influences them, the answer to all these is: "you don't know, and you will never know". You can probably expect no more than 1-2% packet loss for most "normal" situations, but there is no guarantee for that.
Routers may favour one protocol over another, so if someone starts pinging a nearby server and the router is already a bit short on buffers, it might just drop your UDP or TCP traffic in favour of the "more important" ICMP. Your hoster might give TCP a higher weight, and some companies/universities might not forward UDP to/from the internet at all. You just don't know.
Quote:Original post by ZouflainI believe you can detect this pretty easily. Number a bunch of UDP packets and send them to a target location you care about. On the other side you could look at the numbers and you'd be able to see the gaps in the packet numbers.
--What is the average rate of packet loss?
Quote:Original post by SirisianThe situation is unfortunately more complicated than that, and you would need to make many such measurements to different destinations to determine some sort of useful average. However, there are certain conditions which will vastly increase package loss - here are a few I have observed:Quote:Original post by ZouflainI believe you can detect this pretty easily. Number a bunch of UDP packets and send them to a target location you care about. On the other side you could look at the numbers and you'd be able to see the gaps in the packet numbers.
--What is the average rate of packet loss?
Last mile internet access: WiFi, dial-up, IP-over-power-grid and satellite all have much higher packet loss than residential/business DSL or cable.
Distance: Talking to another continent can be terrible, or it can be fine. Probably depends on the exact route taken between (you can find maps of the major cable routes).
Congestion: I can tell when everyone in the area returns from work - my packet loss often doubles. Some areas/ISPs seem better at handling this than others.
I've heard that the average packet loss over the internet is in the region of 1%. The problem is that packet loss tends to be bursty, you will probably alternate between times when you have almost no loss and times where you have lots of loss.
TCP tries to be responsible, it assumes that it is contributing to the problem if it detects packet loss (it assumes packet loss is congestion, which is reasonable for wired networks). It reduces its send rate to compensate (effectively, TCP isn't rate based).
UDP is too raw to support that kind of feedback, though you can add it yourself if you want.
Oliii has an important point: if your program cannot take advantage of out of order or missing packets, then don't bother using UDP because TCP will probably beat you.
TCP tries to be responsible, it assumes that it is contributing to the problem if it detects packet loss (it assumes packet loss is congestion, which is reasonable for wired networks). It reduces its send rate to compensate (effectively, TCP isn't rate based).
UDP is too raw to support that kind of feedback, though you can add it yourself if you want.
Oliii has an important point: if your program cannot take advantage of out of order or missing packets, then don't bother using UDP because TCP will probably beat you.
TCP has one gaping flaw that makes it impractical for a real-time game though and its this. When a packet is lost there is a ridiculously long wait before requesting a resend. if you use UDP, sequence numbers, and periodic heartbeat messages with the last sequence sent and received you will not have the long waits that TCP has and the sequence numbers will easily detect out of order messages. As to the idea of using both a TCP and UDP socket that will half the maximum connections possible on a server. If you are writing something peer to peer that is not a big issue but if its client/server then that is huge.
Quote:Original post by rip-off
I've heard that the average packet loss over the internet is in the region of 1%.
If the last part is WiFi, I'd expect that to rise significantly, and fluctuate quite wildly. When I worked on telecoms testing it was quite possible to see a 50% drop in received signal strength just by standing directly between the 2 end points.
An interesting parallel to this is if the endpoint is reached over a GSM/GPRS connection, as they can adaptively reduce packet loss by spending more bandwidth on error-correction bits, so your problem would manifest as a drop in throughput rather than missing packets.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
