There are two hard things here:

1) Reverse engineering the particular game protocol that you care about. While many games use the same few network protocols at the bottom (RakNet, Lidgren, Unreal, etc) all of the interesting bits will be specific to each game, including payload information -- many games will try to pack bitfields into unaligned bytes, so reverse engineering isn't super simple.

2) Getting anyone to care. If a developer has a problem with their game servers, they will presumably fix that problem themselves (i e, range checks, etc.)

If the game also does NAT punch-through/introduction, then your interposing yourself may add another layer of complexity, as you will need to basically do your own DNAT in a way that still makes those protocols keep working.

A more interesting aspect of network protections to delve into might be DDOS prevention. DDOS mitigation is both really hard, and requires a very low-level view of networking.

There are quite a few companies selling DDOS mitigation services for video games (minecraft was an early favourite), and apparently it's a fairly lucrative field.

Quote

A more interesting aspect of network protections to delve into might be DDOS prevention. DDOS mitigation is both really hard, and requires a very low-level view of networking.

That has almost nothing to do with the game protocol, and almost everything to do with how your network is provisioned.

Typical DDoS mitigation vendors have large "scrubbing" data centers with terabits of data up/downlink. When you are under attack, you will arrange for data intended for your game/site to instead go to their scrubbing centers, they scrub out attack traffic from real traffic, and forward the traffic to you using some kind of tunneling.

In a good world, you have your own BGP for your network edges, and can swing the BGP for whatever network is being attacked to the scrubbing center. This can be done in significantly less than a minute. In a more simplistic world, you have to change your DNS to go to their servers, which will take 15 minutes and up; sometimes way up from that, depending on how other parts of the internet set up DNS caching. Also, DNS protection may not help if the attack is by specific IP instead of name.

A third option is to get as many servers with public IPs as you can. A DDoS will typically be targeted at a few IPs that you own (perhaps based on the physical locale of the attacker,) and anyone using another IP or an IP in another data center will be unaffected.

Now, for games, most of the DDoS scrubbing vendors aren't particularly savvy on how to filter good packets from bad. Thus, if you can simplify the work to profile and write rules for "good" packets for a particular game, and provide those rules to the DDoS mitigation vendor, in some format that they can use, that might be worth something. A sophisticated attacker might similarly profile your traffic and make it look legitimate, but most DDoS-es are more pedestrian, and some amplification attacks (DNS, NTP, and so forth) have to use a specific protocol, which can typically be filtered by a vendor either by port, or by known signature.

If I hooked an API like that into a public OpenArena server that I ran out of my house and tagged it 'honeypot' I'm sure I could learn a lot about DOS attacks and vulnerabilities for that game.

That way, you will only learn about DDoS-es if you somehow invoke the ire of someone who uses a rage booter, and knows your IP address.

Or if you rent some botnet to attack yourself, of course :-)

For passive sniffing, tcpdump and various router flow capture protocols are often useful for diagnosis. I'm not sure how "deep" you want to make the tracing information, because the volume and rate is often bigger than what you'd want to generate over HTTP. There's a reason games use UDP -- it's a lot more lightweight! Doing 60 UDP packets per second per player for a 30 player game is no big deal for a small server slice; doing (60*30) HTTP API calls per second is a lot more work (and serving those same API calls on the back-end where you dump them into whatever stoarge is even more.) Batching would help, of course.

It really comes down to what it is you want to learn about these connections.