Unreal Networking design questions

Started by
17 comments, last by Angus Hollands 11 years, 1 month ago

Well, when I started looking at logic usage that started to be the largest factor. If you have 6 players and 100 Actors in total, thats 6 * 100 checks per update. Which is a lot if its dict checking etc.

It doesn't matter how many characters you have - whatever extra work you do checking which values need to be sent should always be outweighed by the actual sending. If that isn't what you were seeing, then I suspect you were either doing something very strange or testing under quite odd conditions.


In terms of creating a new actor, what happens if a client joins after the actor is created?

When a client joins, you send them all the actors that they need to know about.

I'm used to just having some check on the other side to see if we have the player registered otherwise we create it.

How can you create the actor without having all the information necessary for them yet?

Another question about reliability. can I confirm that whenever a reliable packet is attempted to be sent to a peer, it stores it in a reliable buffer, and checks the ACK packets to see if it got there, else it will resend it. If there is not enough bandwidth, it will ramp up the netpriority until it forcibly enters the outgoing queue. How does this deal with out of order delivery?

This isn't strictly a question about the Unreal networking system, is it? How best to implement reliable messaging over an unreliable transport
is best asked in another thread. You do need to ACK whatever you receive though, because you don't want the other side resending messages you already have.

...but at the cost sending three floats every network tick to the server.

Three floats per network tick is nothing. It sounds like you're prematurely optimising.

Does anyone have any ideas if this is a better method? It seems like I've been thinking about movement code wrongly and this is actually how most people do this. I would also send the time taken to process the inputs, but this of course opens the possibility for speedhacking. Are there better solutions?


Why is it then that bNetInitial is only set False when it receives an ACK? Because if its reliable, the initial attributes will eventually get to the other side? It seems contradictory to me.

Presumably the engine needs to be sure that the data has arrived before it can move forward with other logic. Exactly why, I don't know.

Advertisement

For client side replication: If we receive a replication "packet" from the server, it is denoted by the ID of the entity replicating, and any replicated variables.

If we don't have the ID instantiated locally on the client, we create it and set the replication data.

Of course, I would have to have the server send the full replication because it wouldn't send any previously modified values.

You mention actors that exist already in the level. This isn't a concept that has really applied to me as yet, because I have not considered a scenario where I would want level actors that aren't added depending upon the client connected. Could you possibly elaborate? I saw mention in the Unreal source but I wasn't sure when this would be useful to me.

The question reliability was specific to Unreal (or Tribes) with net priority; I was asking if that is the correct understanding of their model. Out of order delivery was asking how Unreal handles it (although this is related to the typical "How do I order an unreliable protocol"! I wouldn't mind simply dropping out of order packets except when there are delta variables (e.g the difference vector between two velocities) for more precision, which require access to the last state. Additionally, this does not seem to be how Unreal handles it.

My other vague question was about the age of variables. It seems as though there is no specific support for determining the time for which the attributes were valid. This is fine but for extrapolation where I need to know the age of the position and velocity vectors to be able to use EPIC. I think that It would make sense to be able to lookup past states at a particular game tick (a synchronised concept of time) so should I implement this within the replication system, so the networking layer has its packet sequence number and the replication also sends the game tick with the replicated data on top of that protocol?

Let me add some explanation as to why I appear to be so concerned with premature optimisation. I aim at sixty frames per second game frame rate. That leaves 16ms for all processes to execute in. Assuming 1ms is used for the render pipeline on average, that leaves 15ms. That's a lot of time if you write good code, and typically you'll have a fraction of a ms per client. However, this scales geometrically (I believe) with connection of new game-observing clients, as they have to be sent all other actors. Therefore .2 ms may seem small, but when that starts growing with just the scraper logic that seems dangerous. This said, these numbers are merely demonstrating the idea and are not representative of the actual model and this is without relevancy checks.

Seeing as you mentioned your own progress, do you allow any non-actor classes to use replication? or have them handle their own data (e.g the scoreboard, clock)

My final question so far is should the server store the state of all actors every tick? I cannot think immediately of why it would save for replaying, but does Unreal store the "state" of each actor in a lookup by game time? And if so, is it every game tick or every network tick? It seems now would be the time to think about how that would be done, not further down the line.

And thanks for the example source, I'm sure it will be very useful!!

You mention actors that exist already in the level. This isn't a concept that has really applied to me as yet, because I have not considered a scenario where I would want level actors that aren't added depending upon the client connected. Could you possibly elaborate? I saw mention in the Unreal source but I wasn't sure when this would be useful to me.

Actors that already exist on the client would be those that are added to a level in the editor, which both the server and client would have a copy of in Unreal games.

The question reliability was specific to Unreal (or Tribes) with net priority; I was asking if that is the correct understanding of their model. Out of order delivery was asking how Unreal handles it (although this is related to the typical "How do I order an unreliable protocol"! I wouldn't mind simply dropping out of order packets except when there are delta variables (e.g the difference vector between two velocities) for more precision, which require access to the last state. Additionally, this does not seem to be how Unreal handles it.

I recommend using RakNet if possible because it can handle reliability issues for you. Unreal compresses its vectors send sends the full value over the Internet, not a delta from the last state. This way the packet can be ignored if it arrives late or not at all.

My other vague question was about the age of variables. It seems as though there is no specific support for determining the time for which the attributes were valid. This is fine but for extrapolation where I need to know the age of the position and velocity vectors to be able to use EPIC. I think that It would make sense to be able to lookup past states at a particular game tick (a synchronised concept of time) so should I implement this within the replication system, so the networking layer has its packet sequence number and the replication also sends the game tick with the replicated data on top of that protocol?

Unreal handles player movement separately from the rest of its replication using RPC. See this section: http://udn.epicgames.com/Three/NetworkingOverview.html#Player Movement and Prediction

Let me add some explanation as to why I appear to be so concerned with premature optimisation. I aim at sixty frames per second game frame rate. That leaves 16ms for all processes to execute in. Assuming 1ms is used for the render pipeline on average, that leaves 15ms. That's a lot of time if you write good code, and typically you'll have a fraction of a ms per client. However, this scales geometrically (I believe) with connection of new game-observing clients, as they have to be sent all other actors. Therefore .2 ms may seem small, but when that starts growing with just the scraper logic that seems dangerous. This said, these numbers are merely demonstrating the idea and are not representative of the actual model and this is without relevancy checks.

This is why you need to be conservative in determining which actors in a game need to be replicated and how often they need to be replicated.

Seeing as you mentioned your own progress, do you allow any non-actor classes to use replication? or have them handle their own data (e.g the scoreboard, clock)

In Unreal, the score board would be send to the client in WorldInfo which is a special actor representing the game world. You could also simply not use replication for things like that.

Right now I have 3 modes of communication in my game:

Normal replication happens only between actors and 5 times per second. This is done reliably, but order is not guaranteed and older packets can be ignored by the client. Example:
public override void Serialize(DeltaSerializer s, IReplicaClient destination)
{
	base.Serialize(s, destination);

	s.Write(_collideTiles);
	s.Write(_collideActors);
	s.Write(_blockActors);
	s.Write(_layer);
}

public override void Deserialize(DeltaSerializer s, IReplicaClient source)
{
	base.Deserialize(s, source);

	s.ReadBoolean(ref _collideTiles);
	s.ReadBoolean(ref _collideActors);
	s.ReadBoolean(ref _blockActors);
	s.ReadSByte(ref _layer);
}

Event messages can be sent manually between actors, which is what I use for my complex things like player movement or updating the inventory of an actor. For movement, this is done for every game update, which means 30 times per second as opposed to 5 for normal replication.
private void MoveAutonomous(Vector2 delta)
{
	// Only call this on the client when it is autonomous
	Debug.Assert(Actor.Role == NetRole.Autonomous);

	Move(delta);
	Vector2 newPos = Actor.Position;

	// Save moves so we can replay them later if the server corrects us
	_savedMoves.AddLast(new SavedMove(delta, _time));

	// Tell the server to do the same move and confirm or correct our own
	NetOutgoingMessage message = CreateEventMessage(EventServerMove);
	message.Write(delta);
	message.Write(newPos);
	message.Write(_time);
	message.Write((byte)_direction);
	SendEvent(message, NetDeliveryMethod.UnreliableSequenced);
}
Player messages can be sent through through the player objects that represent clients in a game. The client only has its own local player object that can send to the server, but the server has one for each player logged in the game to communicate with its local counterpart.

My final question so far is should the server store the state of all actors every tick? I cannot think immediately of why it would save for replaying, but does Unreal store the "state" of each actor in a lookup by game time? And if so, is it every game tick or every network tick? It seems now would be the time to think about how that would be done, not further down the line.

And thanks for the example source, I'm sure it will be very useful!!

For each combination of a client and an actor relevant to it Unreal stores the values previously sent to that client. Replication is done by comparing the current value in an actor to the last sent value for each connection. This happens during network ticks only.

For client side replication: If we receive a replication "packet" from the server, it is denoted by the ID of the entity replicating, and any replicated variables.
If we don't have the ID instantiated locally on the client, we create it and set the replication data.
Of course, I would have to have the server send the full replication because it wouldn't send any previously modified values.

So the server needs to know that the client doesn't have the actor, in order to send the full replication the first time around. Which means it's not sufficient to just create an actor whenever you get a replication message, because not every replication message is going to be a full set of data. This is why I always have explicit create/update/destroy messages.

You mention actors that exist already in the level.

If I join a game late, what are you going to send? I need some way of knowing who is playing and what their current properties are.

My other vague question was about the age of variables. It seems as though there is no specific support for determining the time for which the attributes were valid.

You can assume the attributes were valid at the time the message was sent. So timestamp the message.

But there is a good argument for handling movement with a separate RPC system, as Inverness has alluded to. Those values change so often and are so important to games that it's often worthwhile to have a special case for them.

Seeing as you mentioned your own progress, do you allow any non-actor classes to use replication? or have them handle their own data (e.g the scoreboard, clock)

If something has properties that need replicating on a regular and/or arbitrary basis, I designate them as an entity that replicates. Otherwise I'll handle them with an RPC (request for the other side to do something) or an event (notification that something has happened). I wouldn't handle a clock as a replicating entity - latency and jitter make that rather unreliable.

I intend to handle Autonomous movement with an RPC. However "simulated proxy actors" can run using extrapolation between ticks, and so I need to handle that, hence needing previous state access.

Here are my initial ideas about how the system should network data:

  1. Nearly (?) everything that should be able to interface over the network should derive from the Replicable base class

  2. This will establish a network ID on the authority for that actor (server-side for nearly all cases) which is replicated to the non-authorities

  3. A Generic actor network handling layer should exist. This will handle network scraping / RPC invocation. Essentially it will scrape attributes and rpc calls (which are invoked, not collected) and bundle the network ID of the actor calling / sending the data. This is then reconstructed on the other side of the network.

If this is the case, there are some issues I can foresee:

  1. How should I support RPC / replication from actors that are static. Take for example the clock. Both server and client need a clock instance, but the server is the authority on game time. How would you designate a network id here? (Ironic that I answered my previous question relating to pre-existing actors). The idea at the moment is that the ROLE_AUTHORITY assigns the network ID. In non-static cases actors will not exist with that ID and so they will be recreated by the replication system. Yet, in the case of statics, they will exist but they will not be aware of their ID outside of the authority. One method I could consider is simply not implementing it client side. If everything that should be replicated is created on the server and not on the client I would enforce an authority-instantiates principle, and clients would delete all actors, including bStatic. However, in doing this I would leave an incomplete system client side. How would I access game-specific actors (like game rules)? Just check if they exist somewhere? I suppose another solution is to preset the network ID on the actor, so it can be matched. Any ideas on how you / Unreal does it would be most welcome.
  2. I have been asking the same question about global access for some time now. I think I need to break some fundamental ideas I have had from my time spent programming. In Python I have always been averse to using global scope namespaces. globals() repulses me in a lot of ways (it seems like most of the time if you're using it you shouldn't be). I think It is that I prefer direct passing of data (invoking functions etc), The same applies to using a module namespace. Is this wrong? I like to think of the scripts has being totally modular. Before the entry point is called, no instances of any data exist. But, the entry point has its own local namespace (it's in a function) so It usually means I have to pass a direct references to the rest of the functions that require these instances. The reason that I ask this is because when an Actor is created, it has to be told about the network interface instance, which will usually have been created in a local namespace and a reference stored inside an instantiated class.

Questions about Unreal RPC implementation:

Firstly, let me assume that I am correct in stating the following about RPC calls:

  1. Simulated functions can be executed by an actor with any ROLE
  2. Non-simulated functions require ROLE_AutonomousProxy or higher to be executed

  3. Functions can only be replicated between two actors (with a ROLE / REMOTEROLE == ROLE_AUTHORITY). And thus servers cannot broadcast.

Therefore, I ask, According to point 3, replicated functions can only be replicated from actors of the same net ID and with one of ROLE, REMOTEROLE == ROLE_AUTHORITY. So, how is this controlled to prevent cheating? I could just check the source of the call on the server, and match the call address to the address that the server believes controls that client.This would prevent the client from invoking calls on other actors it doesn't control, but it would still be able to call the function on the server actor of its local agent from any other actor in its world. This would mean someone could write a special bot that would invoke a server_move() call when the bot saw the local player, allowing the local player to avoid getting shot as easily. How would one create a secure cheat-proof (at the basic level at least) RPC system? Is this something that is enforced client side?

Separation:

My last, and arguably biggest question is about design. I'm not sure how to structure the system; where to set the entry point; should it be the same for client and server. What rule should I follow to know when to separate or join logic? This is the most difficult question I face, and It is rather disconcerting if not upsetting that I cannot seem to think around it.

Fundamentally, I struggle to understand why you would go to the extra effort of writing an abstraction system that enables the use of the same Actor class in each game mode, and then explicitly define server and client code.

  1. Should I think of Server and Client as fundamentally different, with no common ground
  2. Should I think of Actors and replication as a singleton; providing an easy scripting system for people extending games

The specific issue is with the netmode enum. It seems stupid to have it on the client when you would just create a client flavour with the netmode set already. This seems to suggest that there is more of a factory going on than explicit differences between client server. This applies to the network layer (which has basic protocols and responses, which are only called by certain peers (for example request_connect is only received by a server). Or, should everything make use of the netmode and there is just a single game class (With the conditional... I think I will do this unless you offer advice against it). Any detailed help here would be much appreciated. Much appreciated.

I intend to handle Autonomous movement with an RPC. However "simulated proxy actors" can run using extrapolation between ticks, and so I need to handle that, hence needing previous state access.

Previous state and RPCs aren't mutually exclusive. RPCs are just another way of getting information across the wire - how you store it at the other end is up to you.

How should I support RPC / replication from actors that are static. Take for example the clock. Both server and client need a clock instance, but the server is the authority on game time. How would you designate a network id here? (Ironic that I answered my previous question relating to pre-existing actors). The idea at the moment is that the ROLE_AUTHORITY assigns the network ID. In non-static cases actors will not exist with that ID and so they will be recreated by the replication system. Yet, in the case of statics, they will exist but they will not be aware of their ID outside of the authority.

I don't know exactly what you mean by 'static' here but I don't see any reason why a clock would not work exactly the same as any other entity you want to replicate. If, for some reason, it needs to exist on both sides before communication happens and therefore they have different IDs for it, just have some sort of mapping from one to the other. eg. Clients create a clock proxy which can update the local clock. Or just assign the network ID at some point after creation when the server tells the client what the clock's ID is. You can have local IDs and remote (ie. server) IDs and map one to the other when the server takes control of something. It's a non-issue really.

The reason that I ask this is because when an Actor is created, it has to be told about the network interface instance, which will usually have been created in a local namespace and a reference stored inside an instantiated class.

Sounds like a bad idea to me. Actors don't need to know about networks. A separate object can mediate between actors and networks.

how is this controlled to prevent cheating?

The same way everybody checks security of data from client -> server - look at the arguments to the function and see if they make sense.

The fact that it is wrapped up in a replication or RPC abstraction doesn't change the underlying situation, which is that an untrusted remote client has sent a message which may or may not be legitimate. So you treat it as a request with potentially malicious data and handle it conservatively.

I could just check the source of the call on the server, and match the call address to the address that the server believes controls that client.This would prevent the client from invoking calls on other actors it doesn't control, but it would still be able to call the function on the server actor of its local agent from any other actor in its world. This would mean someone could write a special bot that would invoke a server_move() call when the bot saw the local player, allowing the local player to avoid getting shot as easily. How would one create a secure cheat-proof (at the basic level at least) RPC system? Is this something that is enforced client side?

There's no foolproof way to stop a player from modifying their client to instantly react based on incoming game data. This is exactly what bots are, really.

Fundamentally, I struggle to understand why you would go to the extra effort of writing an abstraction system that enables the use of the same Actor class in each game mode, and then explicitly define server and client code.

The Unreal system is just one way of approaching the issue. There is no right and wrong. I don't do it the Unreal way, partly because it is heavily tied in to having your own scripting system, and also because it is too flexible with many ways of achieving the same goals, which I don't like. Personally I try to just have a shared data model, not a shared behaviour model.

The Unreal system is almost 20 years of legacy code. Lots of it is probably only still in there because it's too expensive to remove or rework it.

Ultimately though you will always have some behaviour that is shared across client and server and some that is not. Deciding how to factor that out is something that all developers face. In a way I am lucky - since my client is in C# and the server is in Python, the 2 are necessarily completely separate. The price is a small bit of extra work, but the benefit is that there is no doubt how it needs to be implemented nor any time lost to considering the architectural issues.

I don't know exactly what you mean by 'static' here but I don't see any reason why a clock would not work exactly the same as any other entity you want to replicate. If, for some reason, it needs to exist on both sides before communication happens and therefore they have different IDs for it, just have some sort of mapping from one to the other. eg. Clients create a clock proxy which can update the local clock. Or just assign the network ID at some point after creation when the server tells the client what the clock's ID is. You can have local IDs and remote (ie. server) IDs and map one to the other when the server takes control of something. It's a non-issue really.

Static is the Unreal term for actors that are not "removed" from the world on startup, and so exist without a network ID (they're not created through replication). By assigning the network ID later, the point is null. I need to have a matching Network ID because that is how the owning peer maps the data to the receiving peer. This should be possible in both directions. I like the proposition of a proxy. Alternatively, I could just go with a static_id and be done with it. That would perhaps make things easier. I don't want to introduce a special case for a specific actor class. I would like it to work for more than one instance. I shall have to think about it.

I think I will work with a registration system for actors, whenever the Replicator base class (or whatever I called it) is instantiated it adds the current class reference to the base class list. I could even add methods to get by type etc.

Do you think I should add a special replication case for Actor references? So that it can map them across the network? Or should I just stick with using their NET ID and allow the client to figure out using the getter methods.

May i ask, is your game object the actor with the network id? if so you might want to rethink that. Somewhere in this post i saw python code with a "proxy" declaration, that seams to be the better idea. Seperate your actual game objects from the networked objects. That way you don't have to worry about special cases like "static" actors. They just exist...have different ids but the actor syncs to it.

It as well saves you a lot of trouble with your local entities.

At the moment It won't be the case. Actors are just replicated instances with methods and data. They can hold a reference to a child object as part of their data but the object isn't itself the actor.

I'm a little unnerved by a dependancy that I've introduced. Allow me to explain.


Nearly everything is replicated using the actor replication methods. This includes the worldinfo class. However, there is a problem. The WorldInfo class is needed by the replication system to know whether to call variable replication methods that pipe the serialised variables to the peer (only servers should be able to replicate variables) and it is also needed for function replication.

I planned on making the static (already instantiated) worldinfo class available by using a class attribute (__actors__) which can store all actors that inherit the replicable base class. This avoids the need for a handler to store the actors, and we can take the base class and read the registered actors from it.

Anyway, regardless of how I map the worldinfo client instance to the server instance, does anyone see any qualms with this. In some ways, it makes sense because replication can still occur to the peer, just not from it, and that would ensure that nothing was invoked until the client knew it was a client...like a setup routine. But if anyone can offer a nicer idea, please, by all means!

definition: game-object = Physical Game-Engine object; mesh, objects can also run gamelogic but I don't like it over python

Going back to the local game-objects, How would someone else handle them?

The method that I am considering is a Physics component. Essentially, it's an optimised way of packing and unpacking position, velocity and rotation (see here http://www.pasteall.org/39896/python) that I could have controlling the mesh. For example, instantiating the physics component as a network attribute like this:


physics = ReplicatedVariable("ObjName")

and this would create an instance of that GameObject for this actor's physics component. When the actor is replicated, it will do the same for the other side of course. Then when you need to have the object update its position you can call various methods on the physics component. (Component probably isn't the best name for it). Whenever the component is updated through replication the gameobject can be updated at the same time.

Unreal has a lot of behind the scenes work that can serialise references to actors, and so forth. I was considering doing this; converting to and from network IDs (it may be more hassle than its worth, but I was thinking that this may be useful for replicated weapons). Do you think that it would be better to recreate actor relationships automatically (which wouldn't take a whole lot of work) or force the user to handle the network id? I think it's a bit of a no-brainer but I may as well ask.

This topic is closed to new replies.

Advertisement