For metrics, you want various kind of parameters for gross trouble-shooting:

- customer class (if you have it)

- session state (not-established, negotiating, lobby, game, etc)

- size of packet

- number of payload messages per packet

- payload message types

- packet direction (to server or from server)

- number of dropped packets detected

- number of duplicate packets detected

- number of reordered packets detected

- measured round-trip latency

- number of malformed packets

In the best of worlds, you chuck all of this at some reporting infrastructure, and generate an online cube where you can slice your traffic across arbitrary dimensions (each of the things above classify packets across some dimension.) This doesn't necessarily need to be real-time, because it's useful for finding things you didn't know about your game. Each individual dimension could separately be real-time, and the drill-down would be relegated to a baked cube, for example.

At that point, you can get reports like "number of packets that are re-ordered, containing the 'fire weapon' payload."

Now, there's a second level of diagnosis, where you can get this data sliced, in real-time, based on specific identifiers. Specific IP. Specific game instance. Specific player. Random sampling. Etc. Being able to have a customer on the line and turn on tracing of that particular customer's traffic is super helpful while debugging.

Another useful feature is the ability to capture and play back traffic, both for analysis, and for actual system analysis. If you can capture all packets that go in since server start, then you can reproduce any server state offline for debugging!

Such a system should also have, ideally, an ability to simulate various network conditions and failures. Examples include:

• Being able to simulate various forms of latency (client to server, server to client, bidirectional)
• Dropped packets
• Partial or corrupted packets (it can happen, even with TCP, The CRC only has so many bits)

Obviously, all these things should work with the metrics that are gathered, to allow you to diagnose and mitigate any issues found.

it can happen, even with TCP, The CRC only has so many bits

it can happen, even with TCP, The CRC only has so many bits

Although, luckily, pretty much all physical transport layers have their own detection/correction codes, that significantly improve the robustness. TCP over a 1/10000 reliable physical link would be terrible. Luckily, typical links have bit error rates much, much lower than that.

you are going to register your port, right?

You're kidding, right? If every custom protocol invented was registered, and ports not re-used, we would have ran out of ports in 1982. (Or earlier.)

And, given that you provide the server, and manage the server, how could another service be squatting on the same port? The only reason that could happen would be if some third-party user accidentally puts in the wrong hostname/port information in some other program. Or they did it maliciously -- this is known as a "fuzz attack."

It does make sense to include protocol version information in your preamble, though, so you can reject clients that are too old. This may not need to be part of every packet -- making it part of the initial credentials packet, and making successive packets just rely on successful authentication, might be good enough.

then you lose them in dozens, not just one

Very true!

In fact, most networking hardware seems to have too much buffering, rather than too little, these days, which leads to all kinds of bad oscillation behavior. (Google "buffer bloat" for examples from the TCP world.)

you wouldn't want to play a game in such a setup anyway

You might not want to, but your users are quite likely to try. (And over a 3G mobile connection. And over satellite internet. And over tin-cans-with-string using a carrier pigeon back-up.)

Guess who they will blame when the game doesn't work? Unless you have a very clear meter for quality, and very clearly shows how the quality impacts the game working or not, they will blame you.