So I've been bull-dogging my way through my last few networking projects; simple stuff I've done to teach myself the basics. And it all works, so thats good, but I'm a novice networker and don't presume that I'm doing things the "right way." I was interested in learning more about what typical Game Packets look like, what the structure is, what kind of security is used, error checking if any, special headers or ID tags, what fields should be included in any generic game packet, etc etc. Also, how do you deal with corrupted packets or corrupted partial-packets that are received? I know, really generic question will probably get really generic (or sarcastic) replies, but even examples of "This is what I did" would be really valuable.

Look at Q8 of forum FAQ.

There's a list of networking libraries with source code and all the extras.

There's simply too much ground to cover here.

TCP or UDP?
How much traffic?
Average client bandwidth?
Number of packets sent?
Latency requirements?
How many updates in game?
How many clients?

The solutions are often very specific to exact nature of the game, so there is rarely one-size-fits-all solution.

The corrupted/partial recovery is once again a highly specific topic. For TCP, there are none, you may just lose connection. For UDP, you can a) discard them and request re-send. b) not acknowledge them, and wait to receive it again, c) use redundancy encoding to transfer several packets overlapping, then use that information to reconstruct missing data, d) do nothing and simply move on.

Well thats the thing, I don't have any specific application; I'm just learning net code. I know its very broad and generic, hence I mentioned that. There is no singly right way of doing game networking, which is why "this is what I did in this instance" is particularly helpful. Its like a case-study to learn from. With a given case-study, I can see what a person did, and why, and learn to recognize in which scenarios a similar approach would be useful. "This is my packet design for my turn based rts" is equally useful to me as "this is my packet design for my 64 player fps."

I guess I'm not looking for "tell me what I need to know" as much as I am "show me what you've done so I can learn from it." Generic libraries are nice, I suppose, if you want to use that kinda thing. And I may, but first I want to learn more about the subject. Its nice to know that I can buy a Honda or a BMW, but I'd really like to know how a car works and why you'd take one over the other, ya know?

I can continue doing project after project and bull-dogging my way through each new implementation, and indeed I will, but thats rather slow going. Viewing what other people have done on specific projects and in particular circumstances allows me to learn from other's experiences and mistakes, making my own education progress that much faster.

As far as the Forum FAQ went, I found Q12 (What should I put in my network packets, and how often should I send them?) of most relevance, but I'd always like more.

I read through those posted in Q12, all of them useful and interesting, before posting here. Does everyone just emulate those packet structures, more or less? I somehow thought there would be a bit more variety than ... six. Individual developers sometimes come up with things that even the big boys miss, ya know?

Besides, not everyone is making Quake or AoE. Some projects are smaller, and require different tactics. :)

[Edited by - Luth on April 8, 2007 2:43:47 PM]

I don't use packets at all. I based the serialization toolkit off boost::serialization (the API), then simply developed the message passing interface.

This way I get full compile time safety and both client and server code are guaranteed to fully understand each other as long as they use same build.

Entire system is then based around one single concept, the message class.

For UDP, it's encapsulated with a few extra fields, and a handful of protocol specific packets (request connection, close connection, ack, message group), everything else is done entirely at application level.

Even more, I managed to abstract entire logic using messages only, that get passed between various services. If service is remote or local makes no difference to logic.

For TCP, the only change made is that each message is prepended with 32-bit int, indicating message length (just in case, some string lists can get big).

So simply put, packets are of little concern to me, I focus completely and entirely on logic. Everything else happens behind the scenes.

base message class:


One of concrete message templates (for 3 parameters)


Example of a message definition in application:


Now ClientIdMessage becomes a concrete class which has 3 named parameters (connection_id(), client_id() and client_name())

Templates ensure that as the messages are passed, changing the type or structure will break the build.

The serialize method in base class, through magic of templates completely abstracts the serialization of messages into packets and vice-versa.

This aproach allows me to write entire network code without ever defining what a message is, and writing of logic without ever knowing what network code does.

Also, due to template based serialization, there is never a case where I'd forget to serialize or de-serialize a variable. It also makes it impossible to add a type which cannot be serialized. All of these cause compile time errors.

Apart from primitive types, any object that implements void serialize( ... ) can be written and restored into one of archives (file, network, database).

Message templates (12 of them, one for each number of parameters) are just a subset of serialization, it's used for other purposes as well.

I also use only packet/stream level encoding, not worrying too much about data size at the moment. When a packet is assembled (out of sequence of messages), it's passed through entropy compression as a whole.

The only extra data that the UDP stream requires is the standard sequence code, that allows me to track lost packets and re-send them.

This aproach can be somewhat daunting at start, since a minor change in some packet can cause entire project to blow up at compile time at most unexpected places. But at the same time, without it, those would remain hidden errors.

This aproach is highly efficient, despite apparent bulkiness. Looking at assembly, the overhead of Message classes is 0 in most cases, in some cases, Message class construction is completely skipped and only the necessary variables get serialized - so despite very clean and reliable design, the performance factor is covered as well.

This aproach is very suitable for loosely coupled systems, where number of different messages can be high, possibly in several hundreds, and where losing track due to changes can happen frequently.

Your approach is unique from anything I've done so far. Having not used BOOST before, I spent the last few hours compiling libraries and going over some of the test cases for the Serialization library. Infinitely useful stuff, for sure!

I want to try mimicking your approach, I think. Reading docs and going through source is well and good, but I think throwing myself into the ring and trying to get a boost::serialize based message structure into my "networking demo app" would teach me a lot. Given that I'd say my comprehension level of boost, serialization, and your "packetless" messaging approach is, at best, 25%, I'm sure I'll have some "newbie" questions for you, if you don't mind, and I'm sure more than a few misconceptions that'll need correcting and incomplete pictures that'll need filling in. However, not wanting to abuse your good will, I'll do my best to figure it out on my own. :) Anything you want to volunteer before I batter you with a barrage of queries would, always, be greatly appreciated.


Does your PARAMETER() macro do anything special? Due to my lack of comprehension, I wrote a mimic one as such:
#define PARAMETER(n) P##n m_p##n;

[edit]
Along the same lines, what does your NAMED_PARAMETER() do? My best guess *was removed because it's simply embarrassing*.

Of course, my implementations seem rather silly to me, and I'm sure you must be doing something more involved. Would you mind sharing?

Just defines for fields and accessor.

Messages are considered to be immutable, so there's no setters.

Note that those macros are multi-line, but this code editor interprets them as well, so it messes up formatting if I include the slashes.

A minor detail on why I chose not to use boost serialization.

Boost s11n stores object signatures as well, so it can reconstruct the objects from stream. In my case, that is redundant. For example, if I store a message that has CreatureEntity * and std::list<SimpleEntity*>, then I don't need these signatures, since I already know the exact instances that will be transferred. In addition, transmitting that reveals too much about protocol. So this info is ommited.

This is also the reason why there's that out of place message_id with some obscure constants. The only place where I do need to know about object types is when deciding which message to instantiate. That parameter helps with that, and is also used for packet definitions. This is why it's not implicitly serialized by message classes, but by an external marshaller in server code.

While this imposes some limits on types that can be serialized, in my case I avoid serializing pointers, it doesn't introduce any major problems - but my serialization is much more limited than what boost provides.

This is done due to possible further enhancement, where numeric types could be bit-packed, requiring templated class representation, and I any type description would void any benefit - why pack an int into 4 bits, when you get 4 byte type description on top of it.

[Edited by - Antheus on April 8, 2007 6:15:51 PM]

I've always just used a generic byte packet like:
clicky
(the c++ bit packet I wrote is at the bottom if you want it, it can easily be expanded to allow for any bit size variable if needed).
Then just used simple error checking on the input from the client.

For a simple game a made a while back I just made one packet instance and used that.

packetClass packet(1024);
loop through game object and do something like object.serializePosition(packet);
That's a crude approach, but it works. Lot of ways to manage packets really.

I just got done writing a byte packet system for a multiplayer flash game I'm making, so I'll probably update the article above with any more findings. Since you probably don't care much about packet optimizations, I won't go on any further. (If you do want to discuss it then say so. You can optimize packets usually a lot if you don't care about giving up small to large amounts of CPU speed).