• Hplus:
  I've considered interfaces before but I wasn't sure how to implement them. Perhaps this is because I've always wanted yo work with singular instances of manager classes. Would your idea be to provide a per class interface for ai functions? Inheriting it or composing it?
• Kylotan:
  Thanks. I think I may well 'see what I come up with'! I am sorry that I continue to ask insatiable questions, but I would like to get my solution more right than wrong first time, after having already tried a few different things.

When serialising the game state, would it make sense to join all entity, team, global data into a single "state" type, or should I dedicate a type for each component?
I used to use a polling event system and it would collate all the parts of the system into one state packet, however here are the problems I see at the moment:

• Adding new system aspects would require modifying the state "event" rather than just adding logic to be invoked (which seems a little clunky, less adaptable)
• Single packet sizes become increasingly large with the game as it scales
• Data prioritisation techniques would be difficult to implement[1]

Firstly, because the event system is really comprised of an event with a poll, send and recv method, It seems almost logical that you would add a special event for each different state member. For these reasons, would I be correct (permissible) to assume that it may be better to remove the definition of a single "state" and instead treat the state as a collection of information that it comprises of? I think separating things out would make it less complex to add prioritisation (see below)

Prioritisation [1]
There are two forms of prioritisation I am referring to here:

• Event prioritisation - Which events must be sent when they are requested, and which can be pushed to the next available packet? I think that in most cases this system would simply determine the order of the events sent, and would rarely interfere with the sending of a packet until the state data as we would say that it is needed most network ticks. Here we could assign certain events a higher priority so that entity packets are sent before score packets.
• Entity prioritisation - Which entities must be sent first, such as vehicles, close players to receiving player. I'm not too sure how I would fit this in with the above [2]

I would like to implement prioritisation. My existing system is quite limited in how it transmits to each user (it sends the entire game state, which is fine for small numbers of players, but not at all scalable). The problem with entity prioritisation is as follows:
[2]Assuming that number 2 is essentially referring to the TRIBES and HALO entity priorities, it would essentially conflict with the event priority. An entity which was deemed low priority and low relevance to the user (such as a door opening on the other side of the map) would be given a higher priority in the packet queue than a score packet which would arguably be more relevant. This leads me to conclude that I shouldn't use event prioritisation, and I should determine a global priority. At first I thought that it would make sense to batch entities into one entity packet, allowing me to save space by sending the packet type once for n entities. In doing so, I couldn't use the event priority and the entity priority unless I did some weird heuristics which would add far too much complexity. As it is unlikely that all entities will have exactly the same priority, I think that I would be forced to have a dynamic event priority - one that can vary per-entity. Here are the problems with this idea, to my mind:

• I would now have to increase overhead to 1 byte per entity. I would have to adapt the current packeting system so that all packets destined for the same tick would be batched inside a container. (I already have thought about supporting this using specified depths of packets [3]
• It sort of moves away from having dedicated state packets to a fragmented event dump.

[3] Here is a rough idea for packet separating. I currently have the idea of a specified (limited recursion depth) packet. Each packet sends (by default) its size. Optional protocol headers can be specified. When the game starts, the network layer is initialised with a certain depth of packets, something like this:

self.packet = Packet(child=Packet(child=Packet()))

The depth there is 3 (upper, secondary, lower). There shouldn't be too much overhead. The structure based off my suggestion above is something like this:

<packet_upper>
    size_with_header = short(n)
    base_tick = int(8910)

    <packet_middle>
        size_with_header = short(n)
        group_tick_delta = char(13)
        
        <packet_lower>
            size_with_header = short(n)
            packet_type = char(0)
            ...
        </packet_lower>

        <packet_lower>
            size_with_header = short(n)
            packet_type = char(1)
            ...
        </packet_lower>
    </packet_middle>
</packet_upper>

• Here the upper container represents the outgoing queue. Its protocol adds a base tick, integer, which is the full tick the packet started being populated. This could probably be excluded because network ticks are unlikely to span more than 3 / 4 game ticks, (and even if they were more it still doesn't save many bytes) so this extra tier increases overhead. Forgetting about this detail for now.
• The middle layer is the queue populated for a given tick. This is needed, and all the packets within this layer were intended for that tick.
• The lower layer is the individual event/interface output on that tick. Unfortunately, it adds some overhead with using a short for the size, although I could perhaps limit packets to 255 bytes because most things (entities etc) will be sent as one event instance and the maximum entity size will be around 20 bytes or less [4]

[4] For players, approximately 12 bytes for the position (which could be limited to a short * 3, equating to 6 bytes), one byte per boolean bitmasks, a few extra bytes for other networked attributes.

When serialising the game state, would it make sense to join all entity, team, global data into a single "state" type, or should I dedicate a type for each component?

I don't have a specific answer for you, but when talking about serialisation or persistence, the real question is never "what should I write" but "how will I read". Reading is the harder problem, because you don't have all the type information yet, just a bunch of bytes. So, figure out what would be the ideal reading process for your data - then work out how to structure the writes accordingly.

You go on to talk about events and packets - I'm not sure why you have joined these 2 concepts. Further up we were talking about events just in the context of separating client/server specific code from generic code, but now you're talking about events as something to do with the networked messages themselves. I don't understand why you're doing this so my answers below may seem strange to you.

It seems almost logical that you would add a special event for each different state member.

I would have one event - StateChanged. The payload would tell me what exactly has changed, and to what value(s).

There are two forms of prioritisation I am referring to here:

• Event prioritisation - Which events must be sent when they are requested, and which can be pushed to the next available packet? I think that in most cases this system would simply determine the order of the events sent, and would rarely interfere with the sending of a packet until the state data as we would say that it is needed most network ticks. Here we could assign certain events a higher priority so that entity packets are sent before score packets.
  
• Entity prioritisation - Which entities must be sent first, such as vehicles, close players to receiving player. I'm not too sure how I would fit this in with the above [2]

For the first point, in practice you rarely ever want to send events out of order. If something is 'eventful' then it needs sending and often the order is important.

For the second point, this is not a prioritisation issue but a frequency issue. There is no point reordering the data if you're still sending it all. A data stream of "ABCDABCDABCDABCDABCD" is not going to be noticeably better at giving you A's information than "DCBADCBADCBADCBADCBA". So you're doing the wrong thing here.

If I were concerned about such an optimisation, I'd implement it this way:

• changes to entities set a 'dirty' flag on the property to mark it as having changed
  
• each server-side observer polls entities periodically to collect changed properties, the frequency of such polling being relative to how important the entity is to this observer, sends a state change message, and clears the dirty flag
  
• Obviously each observer needs its own copy of the dirty flags, which in practice either means you have multiple copies of the world state, one per observer (I know of at least one MMO that basically does this), or you simplify (eg. just have one flag per entity saying 'it has changed' and send all the state - this works for simpler games, like first person shooters where you really have little state to talk of).

The rest of this looks a bit like you overthinking it and getting seriously into the realm of premature optimisation.

"I would now have to increase overhead to 1 byte per entity." - that's ok, most computers can send a million of those per second now.

"It sort of moves away from having dedicated state packets to a fragmented event dump." - Your individual messages can live inside an 'envelope message' which is trivial enough, being just a message that contains other messages. But it's uninteresting.

"When the game starts, the network layer is initialised with a certain depth of packets, something like this:

self.packet = Packet(child=Packet(child=Packet()))

" Why? When a message comes in, check its type and read it. If it's a container for other messages, open it up and call the read recursively. No big deal.

Some of the best shooters out there did an amazing job of literally sending out a flat copy of a single struct to everybody 10 times a second. The fact that you're going through all this complex design to achieve the same effect should be a warning sign, in my opinion.

I think, partially because of the language barrier, I have some undefined terms that I need to justify.
Firstly, It's pretty easy to serialise in Python. Attribute lookups aren't too difficult, and in most cases I'd use class attributes that store such identifiers of attributes.
"Serialising" is a loosely defined term. I am not using conventional serialising libraries such as Pickle or JSON because they provide too much extra padding and information that I do not need. Because all data in this system is built from defined attributes that are available to all parts of the system, it is easy to create a format string that can run the struct library unpack and pack methods.

Events and packets. Well, as I've said, I'm using Python, and Python includes a socket wrapper in the standard library. The nature of this means that there is little immediate concept of anything beneath the socket layer. Simply input a destination and a payload and it will arrive (hopefully) at the other end. If I didn't send have data that needed to be queued, and only had data that was needed to be sent every network tick, then this might be an alright method: I can call a socket.sendto function and send the data, but when sending a reasonable amount of data, it is faster to store it in a bytes buffer and send that in one socket call. (this does increase the damage done by packet loss). Because of this, things that are handled by the library that allow me to read each payload from the buffer as individually sent bytes are no longer applicable. They simply ensure my "container" gets from A to B (hopefully). So, I introduced the concept of a "packet". A packet essentially consists of: Default header: size in bytes, Optional "protocol" header (defined by user): packet type, Packet payload. Each packet can optionally nest a sub-packet as its payload. There is no point to allow recursive unpacking because it will always be a fixed depth, and you would need to access the protocol from somewhere. Omitted from my sample code, the nested packets would have a protocol argument which accepts a protocol instance with read/write methods. A depth of three would do the following:

• Uppermost layer - contains the full integer game tick that the packet started "filling" from
  
• Centre layer - contains N entries of tick-container packets: packets which have the delta tick since the base tick determining their creation tick
  
• Lower layer - contains the event data for each game tick specified in the centre layer .

This is all represented by the XML diagram. Furthermore, using automatic detection for nested packets would add to overhead and packet size. (Overhead required to find the protocol for the packet, and then extra bytes to add a type for the packet (e.g 0 = EmptyContainer, 1=MiddleContainer ...). However, as I state this, I would be interested to see how much overhead. It would still make the process more convoluted though.

Prioritisation and state
I am trying to write a system that allows for some deviation from the FPS genre. It's quite easy to do in a lot of ways, save for some genre specific stuff.
Firstly, I intend to have automatic attribute flags. In other words, when the entity class is created:

• It reads all the attributes that are networkable, and sorts them by type (and name within type).
  
• It saves the ordered names of these attributes to a defined variable.
  
• It creates a bitfield for any booleans to pack them more efficiently. Each boolean class is registered with the index and the bitfield that will store their values, and their set and get methods of the descriptor will point to that value.
  
• It creates a format string for struct using the type of the initial value for each attribute, When booleans are found to exist, it adds a Char formatter for each 8 bools. The format string then removes booleans and uses the boolean bitfield(s). I also prepend a contents bitfield (or two) which indicates which attributes are "dirty". The dirty flag is set whenever the attribute is set.

In writing this, I intend to demonstrate that dirty flags would be relatively simple to implement. Whenever the properties are changed, we just read the dirty bitfield, and if it is > 0 it needs updating. At present, updating would require sending all the data, but I could modify it so that it can dynamically send different parts of the attributes

The reason that I am going through all of this "complex design" is twofold.
Firstly, there are few resources implemented in Python. There are basic parts in C++ that are simply not reproducible with the same simplicity as C/C++ and in some cases it would be slower to try and do so. Furthermore, I want the system to be flexible. I already have a working game, but it is very chunky. It doesn't feel clean and I don't like that. For example, at the moment I cannot set "dirty flags" automatically. The entities house the to/from bytes methods, and they don't use networked attributes, instead they just scrape the data. This makes a client entity and a server entity very different, with little code in common (when it really should be in many parts).
As well as this, I want to do things correctly. I've been learning about authoritative networking for under a year, and after having rewritten things a number of times, I want to do them the best way I can. I feel like it is more appealing to a C++ workflow than Python in a lot of ways. I'm not sure where this underlying feeling comes from. Perhaps it is the ease of which one can do something badly and have it work.

There are a few more things I need to ask.

1. Generically speaking, I have two options regarding the client. Firstly, I could attempt to run the same logic as on the server, but using server corrections. This would include all AI calculations etc. The other method is just to extrapolate data. The second method would be harder to do automatically, and It may be best to add some form of manual defined extrapolation that I can easily hook in without modifying the system too much. I suspect that the second option is the one I should choose, simply for calculations sake. Which is the most commonly used for an FPS? (Please note that I have used these words a few times before, and it is for the reason that I don't believe you can improve upon someone else's ideas until you know why they need improving. By implementing conventional methods, you are provided with some idea that it will work effectively, and then you can consider a new iteration solving bugs!)
2. Accessing utilities. How do you share references to instanced classes in C++? If I instance a SoundManager in Python, I still need to find a way of getting it from its namespace to other instances potentially separated as far as it would be out of scope. (E.g self.parent.x.y.soundManager is a little out of place). One cannot simply pass a named reference because If one wants to extend the class with inheritance, it would still point to the old BaseClass name, and you'd be forced then to use classes and not class instances. I just fundamentally hate this part of Python, because there are lots of ways around this (module attributes, special sharer modules etc...) but all seemed wrong in some way.

It's ok, I know all about Python - that's the main language I use most days. I wrote an MMO server in Python (although admittedly it's never had a Massive number of users). And I understand everything you're saying about sockets and packets - I'm just telling you that you're blurring a lot of aspects together which makes development more complicated. The fact that you have a low level socket library to work with shouldn't be affecting the way you handle in-game events or the division between client-specific code and generic code, and the buffer stuff you talk about is irrelevant to pretty much everything else in this thread. And there's no reason why having the serialisation inside the entity means that the client and server entities are very different. Nor is any particular method really language dependent (although pretty much anything you do in C++ could be more easily done in Python). You're letting all sorts of little implementation issues affect your overall design which is going to bog you down.

You say you want to do things correctly - and there is no 'correct'. Just different choices people have made. You ask what the most common method for FPSs is, and I don't know, but I do know that several successful shooters have had significantly different approaches. For example, when it comes to what data to send, Quake 3 just broadcast all state all the time, Unreal has individual attributes marked as replicated across the network, and Torque is somewhere in the middle, where you can divide the state into 32 pieces and send whichever pieces you need. And how to handle prediction, interpolation, and extrapolation also varies from game to game, especially when it comes to resolving inconsistencies. But that wasn't what we were talking about before so it's worth you asking a new question if that's what you want to know about. (Or reading the other times that sort of question has been asked in this forum.)

The same goes for "how do you share references to instanced classes in C++?" That's off topic for this thread so you're best off asking a new question about that. The best way of passing objects from one part of the program to another is just a generic programming issue really. (And did you really mean C++? The rest of that paragraph implies you meant Python.)

After spending some time trying to find the optimal solution to a generically abstracted interface I decided that It wasn't worth it. Instead of trying to treat remote and local entities as the same (which, they're not) I have decided to follow the ideas from UDK's networking. After some thoughts it seems pretty logical. Some questions I have about UDK;

• How are inputs sent to the server? It seems this is handled in the actor, not separate from it which is what i would like.
• I assume that a single actor class exists for client and server, using roles to distinguish between the two. If so; How do certain aspects of the game loop differ between server and client? For example, assuming that actors have an update function, is it just separated into the server update and client update depending upon the role? Or is it the same and just the method it invokes are different.
• Also, some implementation preliminary ideas: I'd like to use bitwise conditions for attribute replication. They'd be faster to compare than lambda functions, and I can't really see many cases I would need to have a lambda function. Would this be a bad idea at first glance? I can always add the option to provide a callable condition.

It seemed unusual that ROLE checks occurred in replication until I realised that in UE2 it was for function replication. So that's easier now. Here's my short progress if anyone is interested. It runs relatively fast, all considered. I still need to determine how to handle position and so on. http://bpaste.net/show/75355/

These are all good points! It does seem to run slow(er) than I'd like, but perhaps It's because I'm running 60 logic ticks per second. In reality, I'd only do such operations every 3/4th tick.

Speedups would come with writing some things in C as extensions :)

Py2vsPy3 Is really, really annoying with libraries, though well maintained and non-hacky libraries can be ported with little difficulty.

In practice I've rarely seen a problem with Python performance, but that usually comes from careful use of the libraries which are often implemented in C. The one time I did have performance problems with Python was implementing A* search, where the standard data structures aren't quite as efficient as the C++ equivalents. Totally agreed on the GIL issue although with MMOs I have always favoured a multi-process/distributed approach with single-threaded simulation in each simulation server, so it's not been an issue for me.

Angus, if you want to know about UDK I suggest posting a new thread about that with a specific title to match. Most people still reading this thread won't know anything about UDK and anybody who does know UDK won't know it's relevant when they scan the first post. Make sure you have read all of the Unreal Networking page and see what questions you still have.

Thanks Kylotan. I have started a new thread, in the API section because most of my questions relate to the logic loop of Unreal and not actual multiplayer concepts. To be honest the reason I continued posting here was because the input that I was most interested in was that of the previous posters to this reply. So, i had little reason to ask anywhere else!

I have been digging apart the networking page, but there are still some points that I need to clarify, hence the new post.

Thanks everyone for your time.