Hey everyone.
I'm redesigning my multiplayer architecture to work in a more manageable structure. At present, client and server classes are separately defined and ultimately share little in common. Simple functions that interface with additional data must be separated for the sake of perhaps one function call, and it is explicitly serialised. Anyway, this is mostly network terms. What I wish to understand is how Unreal (the solution I'm building off) handles the entry points for server and client routines.

For the most part, there is little difference between how the server and client actors run; certain functions are suppressed because they're not client-side or server-side, and certain functions are simply invoked at the other end. However, the fundamental difference exists somewhere in the loop. I understand that both the Server and Client have predefined events that are called onCollision etc.

My first question is how are server and client-side functions separated if they both need to be called but do different things? For example, collision events would do different things on the server and client, If the client-side actor was autonomous it would play a sound before the server called playSound, but on the server it would tell nearby actors but the current actor to hearSound. How is this achieved?

Secondly, how are events sent to the server? It seems like they are sent in an RPC after the client's autonomous actor runs the simulation client-side. But if this is the case, how does the simulation get called? I know of a playerTick function, but how does this choose which function to run? Is it a simple if-switch? If ROLE == ROLE_AUTONOMOUS server_func() else client_func()?

Lastly, simulated functions. I know that the question has been asked many a time, but I would like to clarify their purpose.

I'm writing this solution in Python, so I have a lot of duck-typing possibilities I can exploit. I intend to use decorators to define when functions should be called on the remote side (@remote(world_info, target=SERVER)). I therefore assume that any function without a remote call can be called by either side. I think once I clarify the above points I may answer my own question, but until then what does the Simulated keyword actually mean for the client and server?

Many thanks for your time,

Angus.

My first question is how are server and client-side functions separated if they both need to be called but do different things? For example, collision events would do different things on the server and client, If the client-side actor was autonomous it would play a sound before the server called playSound, but on the server it would tell nearby actors but the current actor to hearSound. How is this achieved?

http://udn.epicgames.com/Three/NetworkingOverview.html#Function call replication

Each program knows its current role, whether as server, client, etc. Certain code only executes when that program is running in that specific role. Servers know to send off function execution to clients if the function is marked for the client.

Just below that part, you have: http://udn.epicgames.com/Three/NetworkingOverview.html#Simulated functions and states

Marking a function with 'simulated' means it's only executed on simulated proxies - on all other roles, it's skipped. It's not strictly about clients and servers - it's about entities on a client that are not controlled by the client. So the 'simulated' functions don't usually make decisions about gameplay, just about presentation (eg. graphical and sound effects).

So basically it's just 2 simple things:
- mark parts of the code as server or client only
- ensure the server sends appropriate messages to the client whenever it encounters some client-only code, to ensure the code is called

In terms of implementation detail, Unrealscript can basically just generate a remote procedure call for each time it sees a function that needs calling on the other side.

The key thing is that there is flexibility in how you implement it. You could choose to have 1 big function for each event that calls out to other smaller functions, and those smaller functions may or may not be implemented on the client or server. Or you might split it up so that you call one client function and one server function, and only the things in the client function will get replicated. There's no right or wrong because it's just 2 routes to 1 goal, ie. getting different bits of script to execute in different places based on process roles.

I merged the topics so that my answer came into this thread; hope that's ok for everyone.

Just to recap and to keep it simpler than my previous answer: UnrealScript handles the division between client and server code by running the same script on all machines but by annotating certain functions so that they only run when the role is appropriate. This is fine for them since they have their own scripting language and can fairly easily implement such a thing.

In Python it would certainly be possible to decorate functions so that they only run when executed under a certain role, so that aspect can be reproduced if you like. The other thing to do is have a decorator able to package up a remote function call and send it off (eg. when a server needs to tell a client to execute something).

The final part is to be able to accept incoming remote procedure calls and execute them. This would not form part of the normal update/tick script but would be done in an event-handling section before or after that.

More info on this is up on Unreal's site: http://udn.epicgames.com/Three/FunctionReplication.html - in fact it's best to read the whole of their networking and replication docs if you want to implement the system their way: http://udn.epicgames.com/Three/ReplicationHome.html

Thanks to everyone. I've considered all of the points. I have read the entire documentation countless times, but certain things were not clear. I'm still not fully sure how best to define replication conditions. I don't know If I will want to evaluate a condition to determine which variables to send every time I want to replicate variables because for any serious number of clients this will become slow.

I have some ideas regarding speeding up the system. All of this is conceptual, but in my old system objects would be iterated through again when they were serialised. Instead, I plan to now use the network attributes as part of the state storage (every tick store the state of the object by copying it's network values) and only converting that to bytes every network tick, but doing so in the same iteration.

One question I have; Would it be a worse design to declare replicating attributes by name, or to use a special declare object that looks more like attribute lookup?

e.g:

class DeclaredProxy:
    def __init__(self, replicator):
        self.tags = set()
        self.replicator = replicator
        
    def __getattr__(self, key):
        network_variables = self.replicator.network_variables
        if key in network_variables:
            self.tags.add(key)
        return getattr(self.replicator, key)
    
class Declared:
    def __init__(self, replicator):
        self.proxy = DeclaredProxy(replicator)

    def __enter__(self):
        return self.proxy
        
    def __exit__(self, *a, **kw):
        pass

class Actor(Replicator):
    a = Variable(1)  
    b = Variable(1)  
    c = Variable(1)  
    d = Variable(1)               
               
    e = Variable(1)  
    f = Variable(1)  
    g = Variable(1)  
    h = Variable(1)   
    
    def replication(self, caller):
        dirty_variables = self.dirty_variables(caller)
        is_initial = self.is_initial(caller)
        
        with self.declarer as declare:
            if is_initial:
                declare.a
                declare.b
                declare.c
                declare.d

Here is the yield example. It's likely less overhead, but less pretty in my opinion

class Actor(Replicator):
    a = Variable(1)  
    b = Variable(1)  
    c = Variable(1)  
    d = Variable(1)               
               
    e = Variable(1)  
    f = Variable(1)  
    g = Variable(1)  
    h = Variable(1)   
    
    def replication(self, caller):
        dirty_variables = self.dirty_variables(caller)
        is_initial = self.is_initial(caller)
        
        if is_initial:
            yield "a"
            yield "b"
            yield "c"     

The time it takes to have a CPU decide whether to send a variable or not is always going to be dwarfed by the time it takes for the network adapter to put it onto the network. So bear that in mind when worrying about what is efficient and what is not.

As for your code examples, I don't really understand what the difference is. Looks like you're worried about micro-optimisations. The most important thing is how simple it is for the user to mark which variables must be replicated or not. You shouldn't need any special code in the Actor class apart from whatever marks or tracks those relevant variables.

I do care about the time spent checking attributes for changed values etc, because that shouldn't be the largest limiting factor.

Also, I am a little perplexed after checking the UT3 source unrealscripts. I am looking at WorldInfo.uc and it has a replication block. Until now I assumed replication was for actors only. As this seems not to be the case, I was wondering how it is so seamlessly consistent between different class types.

1. Specifically, I wish to know how it distinguishes between already instanced and non-instanced classes. Take Actors. I would imagine that when receiving a replication packet on the client for an actor would simply be directed to the actor manager, and if the id wasn't instantiated already it would create the actor. However, this cannot apply to non-actors such as WorldInfo which don't inherently have an ID.
2. To achieve automatic replication, would it be the case that every time something inherits from the Replication base class, it registers the channel ID to a special mapping; so that every time something is instantiated of that type it adds its network ID to that map? This means that replication would actually interface with replicated classes by itself, without needing to be performed explicitly for each class.

Continuing on. The main issue that I face is that UnrealScript is built on top of the engine, and so it doesn't need to define entry points to certain functions like collisions and physics updates. Therefore I believe I need to create an interface layer between the various aspects of my Game Engine and the networked "game" that exists separately to this. For example, defining a network role as Simulated will not be implemented as any functional movement code. Therefore I have to define that myself. This means that I will need to keep a history of "states" of the actors for each game tick to allow for extrapolation using EPIC

I was reading into Unreal and I realised that they send the inputs, and expected outcome and predict between confirmation. At the moment I just send the inputs and skew the time so that it matches closely to when the inputs are received. I think that the first method would be better because then it allows me to avoid guessing and get fewer prediction errors, but at the cost sending three floats every network tick to the server. Does anyone have any ideas if this is a better method? It seems like I've been thinking about movement code wrongly and this is actually how most people do this. I would also send the time taken to process the inputs, but this of course opens the possibility for speedhacking. Are there better solutions?

Also, some other questions about Unreal network attributes.

1. There is bNetDirty which I believe is a simple attribute stored on each actor instance and changes when any networked values are changed. If this is the case how do they stop position and orientation from setting it as it seems to suggest it's optionally triggered? As well as this, It doesn't seem to handle specific attributes, instead it seems that if any attributes change, all would be sent if the replication block checks for bNetDirty only. I assume that the system only sends the changed attributes, so is that caught after the replication block?
2. There is also bNetOwner and bNetInitial. These are only available inside the replication block according to the documentation, and so I assume that bNetDirty is available outside. My hunch is that because attributes are reliably transmitted, if they are told to be replicated because they are dirty, they will eventually get to the other side. Why is it then that bNetInitial is only set False when it receives an ACK? Because if its reliable, the initial attributes will eventually get to the other side? It seems contradictory to me.

I do care about the time spent checking attributes for changed values etc, because that shouldn't be the largest limiting factor.

But it never will be. A modern computer can perform 100,000,000,000 operations per second but can typically only write 1,000,000 bytes per second to the network adapter. Deciding what to send if that decision is a simple yes/no as in this case is never going to be the limiting factor.

Also, I am a little perplexed after checking the UT3 source unrealscripts. I am looking at WorldInfo.uc and it has a replication block. Until now I assumed replication was for actors only. As this seems not to be the case, I was wondering how it is so seamlessly consistent between different class types.

I don't own UC3 nor have I seen the source code so anything I say here would just be speculation. But I suspect WorldInfo and GameInfo are subclasses of Actor, or of some other base class that actually permits replication. I doubt there's any hidden magic here.

Specifically, I wish to know how it distinguishes between already instanced and non-instanced classes. Take Actors. I would imagine that when receiving a replication packet on the client for an actor would simply be directed to the actor manager, and if the id wasn't instantiated already it would create the actor. However, this cannot apply to non-actors such as worldInfo which don't inherently have an ID.

It's likely to be a lot simpler than that - when the server creates a new Actor, it tells each client to create a new one of the same type with the same ID. The same goes for destroying them.

Thanks Kylotan.

Well, when I started looking at logic usage that started to be the largest factor. If you have 6 players and 100 Actors in total, thats 6 * 100 checks per update. Which is a lot if its dict checking etc.

In terms of creating a new actor, what happens if a client joins after the actor is created? I'm used to just having some check on the other side to see if we have the player registered otherwise we create it. Would you suggest that when the server evaluates the replication for the new client it invokes a create function on the client? I'm not really wanting to add RPC functionality beyond the scope of the actors themselves (using directed methods mentioned before) so I would imagine the client would simply deduce if the actor existed and if not create it. The reason I asked this question was because I originally believed that the WorldInfo class did not derive from Actor and So I wondered what sort of black magic was taking place to match the instance on the server to the instance on the client. I think I shall let that wait for now. (the replication of world info).

Another question about reliability. can I confirm that whenever a reliable packet is attempted to be sent to a peer, it stores it in a reliable buffer, and checks the ACK packets to see if it got there, else it will resend it. If there is not enough bandwidth, it will ramp up the netpriority until it forcibly enters the outgoing queue. How does this deal with out of order delivery? I've yet to notice any signficant packet loss in my experience, so I've never worried considerably about order of packets, but if you resend a packet it then creates an out of order sequence on the receiving end. Should I drop the packet but ACK its receipt?

Also, I updated my post above to reflect my evolving thought processes! I shall quote it here, if you would be as kind as to read it!

Continuing on. The main issue that I face is that UnrealScript is built on top of the engine, and so it doesn't need to define entry points to certain functions like collisions and physics updates. Therefore I believe I need to create an interface layer between the various aspects of my Game Engine and the networked "game" that exists separately to this. For example, defining a network role as Simulated will not be implemented as any functional movement code. Therefore I have to define that myself. This means that I will need to keep a history of "states" of the actors for each game tick to allow for extrapolation using EPIC

I was reading into Unreal and I realised that they send the inputs, and expected outcome and predict between confirmation. At the moment I just send the inputs and skew the time so that it matches closely to when the inputs are received. I think that the first method would be better because then it allows me to avoid guessing and get fewer prediction errors, but at the cost sending three floats every network tick to the server. Does anyone have any ideas if this is a better method? It seems like I've been thinking about movement code wrongly and this is actually how most people do this. I would also send the time taken to process the inputs, but this of course opens the possibility for speedhacking. Are there better solutions?

Also, some other questions about Unreal network attributes.

Aside from bNetDirty there is also bNetOwner and bNetInitial. These are only available inside the replication block according to the documentation, and so I assume that bNetDirty is available outside. My hunch is that because attributes are reliably transmitted, if they are told to be replicated because they are dirty, they will eventually get to the other side. So, bNetDirty can be set to False immediately after replication because reliability is just a delay in when attributes arrive. Why is it then that bNetInitial is only set False when it receives an ACK? Because if its reliable, the initial attributes will eventually get to the other side? It seems contradictory to me.

Thanks Kylotan.

Well, when I started looking at logic usage that started to be the largest factor. If you have 6 players and 100 Actors in total, thats 6 * 100 checks per update. Which is a lot if its dict checking etc.

In terms of creating a new actor, what happens if a client joins after the actor is created? I'm used to just having some check on the other side to see if we have the player registered otherwise we create it. Would you suggest that when the server evaluates the replication for the new client it invokes a create function on the client? I'm not really wanting to add RPC functionality beyond the scope of the actors themselves (using directed methods mentioned before) so I would imagine the client would simply deduce if the actor existed and if not create it. The reason I asked this question was because I originally believed that the WorldInfo class did not derive from Actor and So I wondered what sort of black magic was taking place to match the instance on the server to the instance on the client. I think I shall let that wait for now. (the replication of world info).

Another question about reliability. can I confirm that whenever a reliable packet is attempted to be sent to a peer, it stores it in a reliable buffer, and checks the ACK packets to see if it got there, else it will resend it. If there is not enough bandwidth, it will ramp up the netpriority until it forcibly enters the outgoing queue. How does this deal with out of order delivery? I've yet to notice any signficant packet loss in my experience, so I've never worried considerably about order of packets, but if you resend a packet it then creates an out of order sequence on the receiving end. Should I drop the packet but ACK its receipt?

Also, I updated my post above to reflect my evolving thought processes! I shall quote it here, if you would be as kind as to read it!

I'm also using Unreal's networking model as inspiration for my own project, though I'm using C# for it.

Anyhow, a few points:

Actor checking: I suggest adding actors that could possibly be replicated (that aren't already provided by the level itself) to a separate list to be checked. You would further eliminate unnecessary checks by allowing actors to specify a replication rate so less important actors need not use as many resources.

And yes, if a client joins, the server would evaluate replication for that client and send it any actors that are relevant to it and they have not already been replicated to it.

As for reliability, I am using the Lidgren library for C# and used reliable sequenced delivery so packets always arrive but old packets are dropped in favor of newer ones.

As for Unreal networking attributes, bNetDirty is set whenever a property is set on an object in UnrealScript, it is a helper variable so you can skip replication if its false. bNetOwner is true when the client you're replicating to owns that actor, and bNetInitial is true when it is the first time you're replicating so you can send variables that only need to be sent once.

My current source code is available, it might be of some help or it might confuse you. ReplicaManager handles replicating my actors through the IReplica interface. And you can look at my MovementBehavior component to see how player movement is handled. I just copied how Unreal did it in that case.