• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
lerno

UDP network layer must-haves?

27 posts in this topic

I've been trying to gather requirements for an UDP network protocol with virtual connections in a client-server scenario. 

 

This is what I got so far

  1. Glenn Fiedler's series: Use a protocol prefix to filter packets without the prefix (no real motivation given)
  2. Quake 3: Handle the case where the player's port may randomly change due to NAT behaviour. I asked on serverfault.com and heard that this may happen in other cases than with old NATs. I notice this is in Enet as well.
  3. Packet sequencing (obvious)
  4. Acks (implicit or explicit), some protocols use timeouts as well.
  5. Some sort of bandwidth / congestion handling (most implementations have this)
  6. Fragment large packets

What's missing?

0

Share this post


Link to post
Share on other sites

7. Good statistics/metrics from the running system, and ideally reported back from the clients for sampling

0

Share this post


Link to post
Share on other sites

7. Good statistics/metrics from the running system, and ideally reported back from the clients for sampling

 

Anything different from a TCP/IP implementation, aside from monitoring packet loss?

0

Share this post


Link to post
Share on other sites

You need more than packet loss to prove you're getting any value out of the UDP, I would assume.

 

Being able to actually track down cases where NAT punch-through doesn't work right, or users try to fake connections, or whatever, is also pretty useful.

 

Actually, I'm not seeing "NAT introduction and punch-through" on the list. That's probably expected from a commercial network library these days.

0

Share this post


Link to post
Share on other sites

The problem is that NAT punchthrough requires an external server as far as I know (you punch the hole in the NAT by trying to connect to that server). If it's for connection between the players directly you're most likely screwed if they aren't using a VPN or something like that.

0

Share this post


Link to post
Share on other sites

The problem is that NAT punchthrough requires an external server as far as I know


Yes, it does! Any UDP networking library that doesn't actually provide at least some support for it is unlikely to be really all that useful these days.
The support could be as simple as "here's the function to call, and here's a sample server you can run on Amazon EC2 or Interserver or whatever, and punch-through will be yours!"
As long as it's actually built into the library.
0

Share this post


Link to post
Share on other sites

Well, I deliberately excluded it as I didn't think it necessary for a strict client-server protocol where the server uses a static IP. But it's useful to mention.

 

In regards to statistics - both TCP and UDP would measure some sort of latency, so that would not be unique to UDP.

0

Share this post


Link to post
Share on other sites

I'm somewhat unsure about what (1) is good for.

 

In order to discriminate non-malicious UDP datagrams from some different protocol, you would in my opinion just use a different destination port number, that's by all means good enough. How many different UDP protocols do you have in use on your server anyway? Surely not 2,000 or 3,000 of them all at the same time -- it should be quite possible to use a port number (or a range of port numbers) that doesn't conflict.

On the other hand, prefixing a well-known 32 bit number isn't a very serious challenge for someone maliciously sending fake datagrams.

 

If anything, I'd negotiate a random per-connection ID and use that to quickly filter out the most stupid attackers. It's still no big challenge for someone who really means to send you fake packets, but at least it isn't totally trivial to circumvent, and it's more or less a no-op to verify (so it's a good initial check to comb out noise).

 

About (6), I'm also not sure. You primarily use UDP to have low latency. If you mean to send bulk data, you use TCP (easier, more reliable, and exactly as fast as UDP). Insofar, "large data" is somewhat contradictory. Large data cannot have low latency by definition.

I would, on the contrary, require that individual messages within datagrams have a size no larger than 255 bytes (serialization layer). This allows you to length-prefix them with a single byte, which is convenient and also prevents some denial of service attacks. If someone really needs to send several kilobytes or megabytes, they can still split up data at a higher level and send a few dozen/hundred messages, which will go into many datagrams.

 

Also, UDP/IP already supports fragmenting natively, in case one of your datagrams should really be too big.

 

(7) as proposed by hplus0603 is harder to get right than it sounds, but it is equally important. Nothing stinks more than "network that doesn't work" when nobody can tell for sure what exactly doesn't work or why. You really really really want good metrics when there are problems, both to make sure your software works as expected (during beta and stress testing) and for being able to tell a user who is complaining whether it's a problem at their end (and what they can do).

Edited by samoth
0

Share this post


Link to post
Share on other sites

I'm somewhat unsure about what (1) is good for.

 

Me too, and like you say, it appears trivial to circumvent since if you're making a DDOS, you're likely to already have used wireshark or similar to determine what a valid payload looks like. Aside from that Glenn Fiedler seems to know what he's talking about in regards to UDP, so I wasn't about to dismiss it outright. It's not really clear from his articles why he put it there.

 

Any suggestions on metrics?

0

Share this post


Link to post
Share on other sites


About (6), I'm also not sure. You primarily use UDP to have low latency. If you mean to send bulk data, you use TCP (easier, more reliable, and exactly as fast as UDP). Insofar, "large data" is somewhat contradictory. Large data cannot have low latency by definition.
I would, on the contrary, require that individual messages within datagrams have a size no larger than 255 bytes (serialization layer). This allows you to length-prefix them with a single byte, which is convenient and also prevents some denial of service attacks. If someone really needs to send several kilobytes or megabytes, they can still split up data at a higher level and send a few dozen/hundred messages, which will go into many datagrams.

 

Both Q3 networking code and Enet has their own fragmentation code, fragmenting things below an arbitrary guessed MTU you can set up. Isn't the advantage that if you do your own fragmentation, then you can be fairly sure (given that you've taken care to select a good MTU) there won't be any unnecessary defragmentation->fragmentation happening anywhere except for the final networking layer. If each fragment is correctly tagged, then it might be possible to avoid wasting time to wait for remaining fragments of an out-of-date fragment.

 

Maybe there are other reasons as well.

0

Share this post


Link to post
Share on other sites

For metrics, you want various kind of parameters for gross trouble-shooting:

- customer class (if you have it)

- session state (not-established, negotiating, lobby, game, etc)

- size of packet

- number of payload messages per packet

- payload message types

- packet direction (to server or from server)

- number of dropped packets detected

- number of duplicate packets detected

- number of reordered packets detected

- measured round-trip latency

- number of malformed packets

 

In the best of worlds, you chuck all of this at some reporting infrastructure, and generate an online cube where you can slice your traffic across arbitrary dimensions (each of the things above classify packets across some dimension.) This doesn't necessarily need to be real-time, because it's useful for finding things you didn't know about your game. Each individual dimension could separately be real-time, and the drill-down would be relegated to a baked cube, for example.

At that point, you can get reports like "number of packets that are re-ordered, containing the 'fire weapon' payload."

 

Now, there's a second level of diagnosis, where you can get this data sliced, in real-time, based on specific identifiers. Specific IP. Specific game instance. Specific player. Random sampling. Etc. Being able to have a customer on the line and turn on tracing of that particular customer's traffic is super helpful while debugging.

 

Another useful feature is the ability to capture and play back traffic, both for analysis, and for actual system analysis. If you can capture all packets that go in since server start, then you can reproduce any server state offline for debugging!

2

Share this post


Link to post
Share on other sites

Such a system should also have, ideally, an ability to simulate various network conditions and failures. Examples include:

  • Being able to simulate various forms of latency (client to server, server to client, bidirectional)
  • Dropped packets
  • Partial or corrupted packets (it can happen, even with TCP, The CRC only has so many bits)

Obviously, all these things should work with the metrics that are gathered, to allow you to diagnose and mitigate any issues found.

0

Share this post


Link to post
Share on other sites

it can happen, even with TCP, The CRC only has so many bits


Although, luckily, pretty much all physical transport layers have their own detection/correction codes, that significantly improve the robustness. TCP over a 1/10000 reliable physical link would be terrible. Luckily, typical links have bit error rates much, much lower than that.
1

Share this post


Link to post
Share on other sites

it can happen, even with TCP, The CRC only has so many bits


Although, luckily, pretty much all physical transport layers have their own detection/correction codes, that significantly improve the robustness. TCP over a 1/10000 reliable physical link would be terrible. Luckily, typical links have bit error rates much, much lower than that.


Yes, and no...

Older data link layer protocols have quite significant error checking capabilities... being from a time when data lines were usually noisy and not very good. However newer data link layer protocols, due to the increased quality of the lines and equipment, have significantly reduced error correction, preferring to defer that to higher layer protocols. Although they haven't completely eliminated it, you usually have fairly simple error checking (like parity bits). But yes, there is error checking on many different levels. Nevertheless, it is something you should be able to test on your platform, to ensure that your code is handling it correctly.
0

Share this post


Link to post
Share on other sites

Protocol prefixes exist for a couple of reasons (neither of which is to block a malicious attacker).

 

One is to easily filter out another program which happens to be squatting on your registered port (you are going to register your port, right?). People should not re-use IANA registered ports, but they do.

 

The other is to allow you to do breaking revisions to your own protocol later, and have that simply filtered out.

0

Share this post


Link to post
Share on other sites

you are going to register your port, right?


You're kidding, right? If every custom protocol invented was registered, and ports not re-used, we would have ran out of ports in 1982. (Or earlier.)

And, given that you provide the server, and manage the server, how could another service be squatting on the same port? The only reason that could happen would be if some third-party user accidentally puts in the wrong hostname/port information in some other program. Or they did it maliciously -- this is known as a "fuzz attack."

It does make sense to include protocol version information in your preamble, though, so you can reject clients that are too old. This may not need to be part of every packet -- making it part of the initial credentials packet, and making successive packets just rely on successful authentication, might be good enough.

Edited by hplus0603
1

Share this post


Link to post
Share on other sites

And, given that you provide the server, and manage the server, how could another service be squatting on the same port? The only reason that could happen would be if some third-party user accidentally puts in the wrong hostname/port information in some other program. Or they did it maliciously -- this is known as a "fuzz DDoS attack."

 

It is more of an issue for LAN play, where broadcast packets become problematic if multiple applications are using the same port. But I've seen lots of cases where companies (including large companies) decide to use a port that is already registered by someone else for some completely different purpose and just set up shop.

 

As to your first question, if you go look at the IANA list, you'll see that I registered a set of ports for all the Red Storm games back in the '90s.

0

Share this post


Link to post
Share on other sites
Both Q3 networking code and Enet has their own fragmentation code, fragmenting things below an arbitrary guessed MTU you can set up. Isn't the advantage that if you do your own fragmentation, then you can be fairly sure (given that you've taken care to select a good MTU) there won't be any unnecessary defragmentation->fragmentation happening anywhere except for the final networking layer. If each fragment is correctly tagged, then it might be possible to avoid wasting time to wait for remaining fragments of an out-of-date fragment.

 

Maybe there are other reasons as well.

 

There are mainly two reasons why you would implement your own fragmentation layer:

1. You have no clue (rather unlikely for Q3)

2. You know that IP does fragmentation but want to avoid it

 

Why would you want to avoid it? There is at least in theory one good reason. IP discards the whole datagram if one of its fragments is lost. Assume you send a 4 kilobyte datagram with a MTU of 1280, respectively 4 fragments. If you do your own fragmentation, those "fragments" are complete datagrams. If one is lost, you still get the other three. Relying on IP fragmentation means that if one is lost, you lose all four.

 

So much for the theory. In reality, you do not lose datagrams at all. Except when you lose them, and then you lose them in dozens, not just one.

 

Losing individual datagrams because of "noise" just doesn't happen nowadays (except maybe on a pathetic low signal wireless, but you wouldn't want to play a game in such a setup anyway). When you lose packets, it's because some router's queue is temporarily full and it discards every incoming packet until it gets a breather, maybe 0.1 seconds or so later. Insofar, there is no visible difference between losing one fragment and losing all of them, as it's the same observable end result either way.

0

Share this post


Link to post
Share on other sites
then you lose them in dozens, not just one

 

Very true!

 

In fact, most networking hardware seems to have too much buffering, rather than too little, these days, which leads to all kinds of bad oscillation behavior. (Google "buffer bloat" for examples from the TCP world.)

 

 

 

you wouldn't want to play a game in such a setup anyway

 

You might not want to, but your users are quite likely to try. (And over a 3G mobile connection. And over satellite internet. And over tin-cans-with-string using a carrier pigeon back-up.)

Guess who they will blame when the game doesn't work? Unless you have a very clear meter for quality, and very clearly shows how the quality impacts the game working or not, they will blame you.

Edited by hplus0603
0

Share this post


Link to post
Share on other sites


Why would you want to avoid it? There is at least in theory one good reason. IP discards the whole datagram if one of its fragments is lost. Assume you send a 4 kilobyte datagram with a MTU of 1280, respectively 4 fragments. If you do your own fragmentation, those "fragments" are complete datagrams. If one is lost, you still get the other three. Relying on IP fragmentation means that if one is lost, you lose all four.

 

Now, I haven't looked in detail at the fragmentation code, but I was assuming that losing a fragment would invalidate the entire packet anyway - i.e. one lost fragment means you lost all, just like with IP.

 

Is there no merit to my conjecture that it can be worthwhile to pre-fragment packets to avoid any reassembly at network boundaries and minimize MTU-related issues?

 

Of course, it might simply be that they wanted to allow packets that exceeded UDP max packet size.

0

Share this post


Link to post
Share on other sites

I was assuming that losing a fragment would invalidate the entire packet anyway - i.e. one lost fragment means you lost all, just like with IP

 

 

I didn't look either, but I doubt it. I would deem myself very arrogant if I assumed that I could write a fragmentation layer that is exactly identical to how IP does it, only better. In all likelihood, it would only be worse! Carmack and the other people involved in Q3 should know better than to try such a thing.

 

Of course, it might simply be that they wanted to allow packets that exceeded UDP max packet size.

 

 

That would be insane, however. If they have that much bulk data, then they'd better use TCP in the first place. UDP already allows you to send datagrams close to 64kiB in size, which is way too big for "realtime".

If you use UDP, you do that because you have hard realtime requirements on data arriving, you want low latency. You just cannot afford to wait. Thus, you certainly do not want to send messages of several hundred or thousands of kilobytes, and you certainly do not want to wait for dozens or hundreds of fragments to come in before you can assemble them.

 

You probably won't die if every now and then a single datagram is getting fragmented, if it happens, well then it happens, bad luck -- but planning to send datagrams several dozen of kilobytes in size is just a bad idea.

Edited by samoth
0

Share this post


Link to post
Share on other sites


I didn't look either, but I doubt it. I would deem myself very arrogant if I assumed that I could write a fragmentation layer that is exactly identical to how IP does it, only better. In all likelihood, it would only be worse! Carmack and the other people involved in Q3 should know better than to try such a thing.

 

From what I understand there appears to be some issues regarding IP fragmentation with different MTU. If I understand correctly then the situation is like this:

 

You send a packet of n bytes where n > MTU of your network. Let's say we send 4000 bytes and the MTU is 1500, so that gets split into 3 packets. At a later point this packet then needs to pass through a relay with an MTU of 1450, so suddenly those two first fragments are too big and the sender may either send an ICMP back with "packet too big" or fragment the packet again.

 

To avoid that, one solution would then be to make sure that each packet is small enough that it's unlikely to encounter any fragmentation until it's assembled at the recipient's.

 


That would be insane, however. If they have that much bulk data, then they'd better use TCP in the first place. UDP already allows you to send datagrams close to 64kiB in size, which is way too big for "realtime".

 

Well, it's likely to be for initial map downloads and such. It makes sense to either use TCP and UDP.

0

Share this post


Link to post
Share on other sites
Guess who they will blame when the game doesn't work? Unless you have a very clear meter for quality, and very clearly shows how the quality impacts the game working or not, they will blame you.

 

Ah yes, I can see that kind of thing happening. But luckily, to your advantage, no game (except turn-based like chess) will really "work" in such an environment, and if you display something like "packet loss!" (or even a number) in the corner to hint them, it should hopefully be good.

 

To avoid that, one solution would then be to make sure that each packet is small enough that it's unlikely to encounter any fragmentation until it's assembled at the recipient's.

 

Problem with a homebrew fragmentation implementation is that you don't really know when such a thing happens. At least, there is no easy way to find out. ICMP makes this work "magically" under IPv6 but you cannot easily access the info from any unprivilegued user process. Fragmentation happens automatically at the router under IPv4. You never know it happened.

 

If you do your own fragmentation, you only have the option of setting the "don't fragment" bit, but it's not very straighforward (for example Linux only allows DF for raw sockets, so you must run as root), and in my opinion it's something that stinks (for UDP, at least). Other than that, your only way of knowing that you've exceeded the MTU is by looking into your crystal ball, pretty much.

 

TCP does the same thing behind your back to discover the MTU and its window size, sure, but that's a different story. TCP is "bulk data", not "real time". It's perfectly OK to have packets dropped purposely to find out your limits.

 

For UDP, I would rather try to keep the datagram size to something reasonable that will likely pass on 99% of all routes, simply by sending less stuff at a time at application level. Something like 1280 bytes should fly (since practically the entire internet is IPv6 nowadays, even though most people use IPv4, the routers still have to comply with IPv6's minimum MTU). Maybe this will cause fragmentation for a few people who have a lower MTU (in theory, they could have as little as 576, but I doubt you'll find a lot of these, I can't even remember a time when it was anything lower than 1492 on my end), but you probably won't need to care. First, it just means that a select few people get 1 datagram in 2 fragments, what a tragedy -- it's not like they're getting 50 of them. And second, it'll be very few people, and those likely won't have any fun playing your game with their low-end internet connection anyway.

 

On the other hand, doing your own MTU discovery will necessarily and regularly drop datagrams for everybody. Which is perfectly OK if you use TCP to download a 50 MiB file. If TCP drops and resends 10 or 20 out of 50,000 packets to discover the best MTU and window size, your 20-second download takes 0.01 seconds longer, if it does at all (most likely it doesn't even) -- who cares. There is no real difference.

Only when it's UDP data in a game, you'd wish that packets weren't dropped purposely. Every now and then, you'll lose packets anyway, and it's bad enough when that happens. Sure, your application must be able to somehow cope with packet loss, but meh. Every packet lost somehow interferes with gameplay, maybe more severely or less so. In any case, I would never provoke it to happen on purpose, on a planned schedule.

0

Share this post


Link to post
Share on other sites


Problem with a homebrew fragmentation implementation is that you don't really know when such a thing happens. At least, there is no easy way to find out. ICMP makes this work "magically" under IPv6 but you cannot easily access the info from any unprivilegued user process. Fragmentation happens automatically at the router under IPv4. You never know it happened.

 

But you're saying that MTU of 1280 should pretty much always work, so why not break your packet into such fragments to begin with? I mean, you say one should try to keep the UDP packet size down, but doing so pretty much would need you to split your packets at application level instead.

 

In some cases that can be better, but then again a custom fragmentation doesn't prevent you to do application level splitting if you want to.

 

What you say seem to support custom framentation rather than the opposite. What am I missing?

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0