Sign in to follow this  
Synergi

Authorative Setup Client Prediction 100% Wrong

Recommended Posts

Hello GameDev Community,

this is the first time, I look forward on contacting you for help, but I really need you as I'm currently totally stuck. Not a single Tutorial Like GafferGames, MPDev, ... was able to explain me what I'm doing wrong and I'm currently really stuck.

The issue I currently face is quite obvious, I know for sure in beforehand that my client side prediction is for 100% wrong. But I need to do it do simulate zero latency for the user. But in the end it is wrong and I have to correct the user. So yeah I'm completely stuck. Something in my network design is really bad to not properly fit Client Side Prediction.

In the game engine I use (and mostly every other game engine), movement is explained in a given speed * time. So let's say Speed is something like 5m/s. So Entities move at an amount that is dependent on the time and speed. The time is very important as this is also a realtime 3D game.

I read most of the good tutorials out there in the wild internet about good authoritative networking but I can't spin my head around client side prediction and server concealment. Please don't misunderstand me, I know for sure that this is needed to simulate zero latency and also ignore the RTT for the change to upon based upon the Input. But:

Every Input that triggers an Action that is depending on the previous one (which are most) will be wrong predicted.

Let me explain this with an example:

Server runs at 10 Ticks per Second. So in between each SnapShot is a delay of 100MS. Let's say the client currently interpolates in between the states 500ms and 600ms. The RTT is 100ms so the ping is 50ms. So as we interpolate on the client side in between 500-600 Snapshot the server is on 650ms.

Now the client sends the button press "W" that triggers the movement forward action. This arrives at the server at 700ms. The client will start running at server time 650ms. The server sends back the ACK of this action with the server time 700ms as this is where the server marks the character to run. Now the client can mark the 700ms world state he will receive with his 650ms Snapshot of his position to now if the predicted state is correct.
Now the client send at 700ms the action "SPRINT" to the server. This triggers double speed. This time the package of the input to server takes not 50ms instead it takes 100ms to arrive. so the Server will send an ACK with 800ms. So to summarize:
 

Client -> 650ms MoveForward -> Server -> ACK @ 700ms (Ping 50ms)
Client -> 700ms Sprint -> Server -> ACK @ 800ms (Ping 100ms)

So on the clients predicted state we run for 50 ms and then sprint.
But on the server state we run for 100ms and then sprint.

This example is just blown up. For a real world scenario this would be more like the ping is 50ms then 55ms then 48ms. So every time the latency jumps around and is never the same. Knowing this, I know for sure, that my predicted states will be 100% wrong. I will have to correct them.

So what am I missing here that I can't wrap my head around this? Most authoritative game has a client side prediction as the latency would be too noticeable. But it can't be that they always correct the predicted state. In fact a good prediction system has the least correction amount.

So please help me to see what I miss x)

Share this post


Link to post
Share on other sites

For such games, I've read that the server can have a window of time where it is willing to apply inputs "in the past". The server must have some level of buffering or the ability to unwind so that it can replay the simulation as if that input had arrived at the estimated time of sending. Subsequent updates will correct the other clients, who will need need to interpolate these updates over time to try to hide any jumps caused by this behaviour.

Note that you can still get undesireable behaviour, which is a fundamental "speed of light" constraint of trying to run a real time simulation with non-trivial latency, but at least now there is a potential happy path where the server and client can agree!

For more details, I'll refer you to the forum FAQ, in particular Q12, Q16 and Q27 seem most relevant.

Share this post


Link to post
Share on other sites

For such games, I've read that the server can have a window of time where it is willing to apply inputs "in the past". The server must have some level of buffering or the ability to unwind so that it can replay the simulation as if that input had arrived at the estimated time of sending. Subsequent updates will correct the other clients, who will need need to interpolate these updates over time to try to hide any jumps caused by this behaviour.

Note that you can still get undesireable behaviour, which is a fundamental "speed of light" constraint of trying to run a real time simulation with non-trivial latency, but at least now there is a potential happy path where the server and client can agree!

For more details, I'll refer you to the forum FAQ, in particular Q12, Q16 and Q27 seem most relevant.

 

Wouldn't this approach destroy the authoritative idea of a server? I mean the client could force a high latency to have the advantage of seeing the past and reacting to it with a rewind on the server. Wouldn't that be a possible attack for cheaters?

Share this post


Link to post
Share on other sites

Yes, it does introduce a risk for that. You have to balance that risk vs having an unresponsive experience for everyone. I'm not aware of any AAA game that has zero cheating, thus I infer that the state of that art for those with the most resources cannot solve this problem.

The server is still authoritative, as it can choose to discard the input if it is "too late" or you heuristically infer that it might be a hack (e.g. this client has a typical ping of 40ms but this "shoot gun" event arrived 150ms late).

Share this post


Link to post
Share on other sites

Ok I can understand the concept of if a package arrives really late I could still give the Client a chance to accept this package by rewinding on the server a bit. 

But how would you take care of packages being sent to the server with a slightly different delay like lets say

Client -> MoveForward -> 500MS (ping 50ms)
Server -> MoveForward ACK -> 550ms
Client -> Sprint -> 550ms (ping 55ms)
Server -> Sprint ACK -> 605

So the Client thinks he is walinking 50ms and then sprinting but on the server he is walking 55ms and then sprinting? I mean I could always slowly interpolate back to the correct state, but pls don't misunderstand me, why should I predict something that I know will be wrong? x)

Share this post


Link to post
Share on other sites

But how would you take care of packages being sent to the server with a slightly different delay like lets say

That's probably less common of a situation than you might think. Round-trip latency tends to be pretty constant over short timescales.

Share this post


Link to post
Share on other sites

 

But how would you take care of packages being sent to the server with a slightly different delay like lets say

That's probably less common of a situation than you might think. Round-trip latency tends to be pretty constant over short timescales.

 

 

So you would say, that if I compare the Snapshot from the Server with the Snapshot the client if the difference delta is less then sigma I Ignore the difference and let my prediction be correct? Or do I correct always and live with a really terrible prediction system?

I mean please do not misunderstand me.

But when you check games like RocketLeague or Smite they have a fast paces 3d enviromet. How do they do such things?

Edited by Synergi

Share this post


Link to post
Share on other sites
I believe the client and server try to synchronize on a shared understanding of time, at least in terms of "number of net syncs", thus packets can be "timestamped", e.g. if you net sync 10 times a second, then each sync is 100ms. I can't recall the details, check the forum FAQ, but if memory serves it involves each peer echoing back each other's net sync counter to sync up.

I'm not sure I understand your notation, for me the interaction, starting at time T, could be (assuming constant latency 50ms):

Client A @ T: Start walking @ T
Client B @ T: Player A idle
Server @ T: Player A idle

Client A @ T + 50ms: Start running
Client B @ T + 50ms: Player A idle
Server @ T + 50ms: Receive walk message, rewind simulation to T, apply walking action to A, broadcast state at next net sync time (assume instantaneous)

Client A @ T + 100ms: Stop moving
Client B @ T + 100ms: Player A position is moving since time T, position at T + 50ms included. Animation starts and client starts interpolating to reduce perceived warping (Player A will move "fast" to catch up)
Server @ T + 100ms: Receive run, rewind to T + 50 ms, apply run, broadcast state next net sync time

Client A @ T + 150ms: Player A idle, final position confirmed by server
Client B @ T + 150ms: Player A is running since time T + 50ms, position at that T + 100ms included in packet.
Server @ T + 100ms

You can imagine now that at T + 200 ms, client B is made aware that player A has stopped at a given location.

The scheme is more or less the same even with each link having it's own latency that varies, but harder to represent clearly by hand - Hopefully I've not made any mistakes as it is!

Share this post


Link to post
Share on other sites

This sounds wonderful with rewinding on the server to the synced timestamp. I did think about this before I tried to go full authorative. But this could be so easily hijacked from a hacker. As an example have 2 characters Race a given distance.
The Hacker could now hijack the timestamp of the package and just instead of saying it was at 500 could change it to 450. Now the Server rewinds to 450 and lets the hacker run 50 ms earlier then the other. The hacker will always win.
On games like RocketLeague or Smite where you have a Grand Prize of 100k I simply don't believe that their system would be that easy to hijack.
I totally understand that such knowledge on howto properly solve such a problem isn't spreaded like the standard authoritative concepts, but there has to be a website that demonstrates how it is done. That is the internet x)

Share this post


Link to post
Share on other sites

Ok I think the best approach currently is to rewind on the server to the time of press and inside the package the client sends to the server containing the timestamp to rewind to, I simply crypt this area of the message or something like that. I can't find any other solution other then rewinding on the Server. 

Share this post


Link to post
Share on other sites

But this could be so easily hijacked from a hacker. As an example have 2 characters Race a given distance. The Hacker could now hijack the timestamp of the package and just instead of saying it was at 500 could change it to 450. Now the Server rewinds to 450 and lets the hacker run 50 ms earlier then the other. The hacker will always win.

Keep in mind that your communication channel is typically sequenced (guaranteed to be so if using TCP), so as long as you enforce that timestamps coming from the client are monotonically increasing, the client can never really execute a one-off rewind attack that way.

Share this post


Link to post
Share on other sites
Encryption doesn't solve the problem, it just makes it a little harder for a hacker to figure out what you're doing.

For your hypothetical racing game, you can reject inputs that are for "before" the race begins. Another point to note is that even without a rewinding feature, a hacker has an advantage as they can estimate the ping and send the "start running" command to arrive the instant the race is due to start.

My understanding of serious e-sports games is that the device is provided by the tournament organiser, at least in the last few rounds where the large prize money is.

Share this post


Link to post
Share on other sites

Synergi, you're overthinking it. Every prediction system is, by definition, going to be a little bit wrong sometimes. There is no solution to that. All you can do is mitigate the problem, which usually means correcting the client a small amount if and when the server reports that its movements were somehow wrong.

Share this post


Link to post
Share on other sites

Networking and game design are closely related and there are always trade offs involved. If you're making a competitive game and the single press of a button determines the outcome (e.g. your race example), then you might want to forgo responsiveness and let the server decide when the button was pressed. If that makes your game feel too unresponsive, you can try hiding the latency with client side prediction/animation/sounds. But you could also go for another design, where the outcome is not determined by a single button press. If you need to maneuver through some track and other players can interact with you, it becomes less attractive to send some input in the past, because it requires the cheating client to not have given any input until that point in time (doing so would prevent the server from accepting the past input since all packets are sequenced) and it binds them to that choice which might make reacting to other events impossible.

For a game like Smite, I can image that movement allows for some rewinding to make that feel as responsive as possible, but the use of abilities may not.

Another point to note is that even without a rewinding feature, a hacker has an advantage as they can estimate the ping and send the "start running" command to arrive the instant the race is due to start.

Ideally you want this to be in the game by default, the client's simulation should run a little bit ahead of the server. The added benefit is that your server will have to do less rewinding and forward simulations, which can be costly (processing wise) depending on the type of game.

Share this post


Link to post
Share on other sites

Not a single Tutorial Like GafferGames, MPDev, ... was able to explain me what I'm doing wrong


It's not the role of tutorials to tell you what you're doing wrong.
It's the job of your own debugging to tell you what you're doing wrong.
Clearly state what you believe "right" is, and why.
Now, add asserts to your code, and breakpoints in the debugger, and print statements, and other barriers that will let you prove where you get it "right" and where you get it "wrong."
Once you see what is wrong, work your way backwards from proven-right points in time to see where you diverge.

Yes, this may require you to add more data "on the side" to verify against, such as logs of "data I reveived at time X with position Y" to look through.
That's fine. You can remove this before you ship. (In fact, typically, you'll have a "develop" versus "ship" code switch to do this.)

Share this post


Link to post
Share on other sites

Server rewinding to account for player's ping spike is a terrible idea imo. If a player has unstable network condition, he/she should not expect a smooth gameplay.

Don't have to be spikes at all, simulations will diverge even with best connections out there. Unless you have lock-step, you either need to have rewinding or continuously correct clients. For some games (e.g. FPS), it's imperative to have rewinding for them to be 'playable'.

Share this post


Link to post
Share on other sites

For some games (e.g. FPS), it's imperative to have rewinding for them to be 'playable'.


I wouldn't say you need server rewinding


I think you mean different things. I think Mussi means server rewinding when doing hit checks, similar to the Counter-Strike model.

Share this post


Link to post
Share on other sites

As Kylotan pointed out to me, the term rewinding is ambiguous and can mean looking up some previous state, and going back to a previous state and simulate forward again to the 'present'. So to clarify, to solve the case of simulations becoming divergent you don't need either form of rewinding on the server. You could rewind and forward on the client or you can just send the correct state from the server for clients to snap/interpolate to.

For some type of games you will need to be able to look up a previous state. Having that could make it trivial to add the ability to forward the simulation from that state, which might be a reason to go for that route.

There are a lot of ways to go about this, each with their own costs and benefits. Choosing what's right for you depends on the time you are willing to invest, the complexity you can handle, the type of game you are making and other requirements you might have.

Share this post


Link to post
Share on other sites

Ok to nail this down, I was doing a lot of research about this topic and didn't find a proper solution. So I designed a mix of both and I share this with you:

1) We use both techniques Rewinding and Correction
2) We mix them in a very clever way

Rewinding has the negative of becoming unfair to other clients if rewinded to far
Prediction can be always wrong if no rewinding is applied.

So both systems have a negative. But we can remove them mostly when mixing them cleverly.

When dealing with such systems you normally have a snapshot interpolation system on the clients to render the world in past for lag compensation. This gives us also other possibilities.

Normally you delay the past for 100ms (depends on your Snapshot rate 100ms delay = 20/s) We can use this 100ms delay.

So we have to set the following rules for our system:

1) If a package arrives and the timestamp is only <50 ms behind we can rewind and resend the state to the clients
2) If a package arrives and the timestamp is > 50ms && < 100ms we can rewind the last state and assume the client is moving since 100ms so we rewind the previous state to have actually moved 50ms and the next is going to be also 50ms moved. Then we resend the previous state
3) If a package arrives and the timestamp is > 100ms we have to correct on the client for the amount Above 100ms. So we combine rule 2 with a correction on the client above 2.
4) The resend states are marked as a resend so they are special states
5) If a client has a action that needs hit detection on the server as an example he sends the stateID with it. If on the server the state was already corrected but as the Client sent the non corrected state we do the collision against the non corrected one to be fair as the other client hasn't received the corrected state.

With these rules we can ensure that our prediction system is 100% correct if delay is <100ms and if we have a delay above 100ms we only have to correct the amount above 100ms. and here we can decide between snapping and interpolation corresponding to amount we are off. Lets say we are only on 110ms delay we could interpolate the last 10ms but if we are on 200ms delay we would snap.

I think the system I created is very well. If you find something that is better of not logic please tell me.

Share this post


Link to post
Share on other sites

This sounds like it just moves the "on time" deadline backwards by 50 milliseconds?
Why wouldn't you just shift the time for which clients send packets forward by 50 milliseconds, for the same effect?

I need an example timeline where you show what happens on the client and server as I don't exactly know what you mean? 

Share this post


Link to post
Share on other sites
You are saying that you re-simulate the server based on data received by the client up to 50 ms late.

How is this different from just saying "the deadline to simulate time X is moved down by 50 ms?"

What you end up doing here is either waiting to send an update to other players until the deadline has passed (so, delay updates by 50 ms,) OR send updates that you know are wrong, and you reserve the right to change the outcome later (essentially, send extrapolated values.)

Why wouldn't you pick one (delay updates more, or send extrapolated values,) and just make that the deadline? It seems like your analysis of "separate cases" just muddies the waters.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this