Clientside prediction / Rewind + Replay bug

Started by
22 comments, last by hplus0603 15 years, 12 months ago
I guess basically what I'm wondering is, if you're only updating physics 20 times per second, how do you keep it smooth?

My game is in the twitch game catagory, requiring precision and low latency.
If you gave a helpful reply, I rated you up.
Advertisement
Typically, you'll do extrapolation/interpolation for the remote entities, and you actually let the user turn faster than the physics rate. Note that players are much more sensitive to mouse lag than to button lag. Also, you can play gunshot animations/sounds as soon as possible, even if you wait 50 milliseconds to actually simulate the effect.
enum Bool { True, False, FileNotFound };
Quote:Original post by hplus0603
and you actually let the user turn faster than the physics rate.


I'm not understanding this part... The player can't get ahead of their own physics can they?

If you gave a helpful reply, I rated you up.
I believe he means the direction the player is facing is updated every render frame, but the direction the player is moving is only updated every physics tick. For example; play quake (or most fps) and pull the network cable... you can't move or fire, but you're likely to be able to look around still.
Ah, I wasn't thinking in terms of FPS games.
If you gave a helpful reply, I rated you up.
I had run into similar problems a few months back when I was developing a small FPS demo.

The following are those I acquired from local perception filter, local player prediciton, quake3 source code to solve the problems:

1) network latency

frame-locked authoritative update from server to client

server: 1  2  3  4  5        \  \  \  \  \ \n         \  \  \  \  \ \nclient:   1  2  3  4  5


------------------------------------------

frame-locked input from client to server

server: 1  2  3  4  5           /  /  /  /          /  /  /  /client:  2  3  4  5


2) playback without prediction

first, you need to buffer the authoritative update sent by server

then you need some sort of methods to determine at what time you should be playing back the authoritative sequence received from server

my method is to estimate what time a given frame is at. whenever you received update from server, timestamp the update (don't use float, double or uint64_t is fine), to estimate, e.g. at what time will frame 6 update arrive:

frame6_time = ((frame1_time + inv_fps * 5) + (frame2_time + inv_fps * 4) + (frame3_time + inv_fps * 3) + (frame4_time + inv_fps * 2) + (frame5_time + inv_fps * 1)) / 5;

so, based on this association (6, frame6_time), whenever given a game time, you can calculate what pair of frames you are going to interpolate and the interpolation amount. if you want the playback be less susceptible by network instability, you may add some amount to the estimated time as such (6, frame6_time + some_amount), the amount may be based on network jitter and a small value, e.g. half a frame interval, however, what you will be seeing is more out-date.

3) input buffering and prediction
input buffering is needed for both client and server

in server, in order to process the input sent from client correctly in sequence, every input from client is frame stamped, and server will never process out-date input, if there is no input of the current processing frame, it will use the last frame's before the current processing frame's. so in order to always have input current to the processing frame, client must send input ahead, the estimation is similar to 2)'s, but minus ping (round-trip delay), jitter and a small value

in client, the purposes of input buffering are for prediction and network traffic optimization. to always predict 'correctly', whenever you receive authoritative update from server about local player, you predict from the server update frame plus 1's input in the input history to the last input frame you sent or you buffered. this is different from your detecting errors with some threshold then correcting, this approach is to always correct the prediction, i learnt it from quake3 source code, you may look into it, the comment said its not so bad, however, for larger network latency, the cpu load for always correct errors approach will be increased (if you are only predicting your local player, it should be fine).

4) conclusion
you need to work out the timing very accurately, don't try and error in the first place, always consolidate your work on paper or your mind and try dry run

in my FPS demo, for more instant interaction reason, server sends authoritative update 30 times a second, each update includes one frame's position, animation, ... etc, client buffers up to 2 input frames or some other critieria before flushes them out, will only buffer input different from the previous frame. physics update occurs 60 times a second. for demo scale reason, client and server are both singlethreaded design, but network is tcp multithreaded iocp with nagle algorithm disabled, v-sync is turned off for steady time estimation, rendering occurs when e.g. current frame should be 61, and last rendered frame is 60. prediction is in total 4 frames ahead at minimum, accounts for minimum 66.7ms latency

hope this helps

[Edited by - hellknows2008 on April 25, 2008 3:49:41 AM]
something occurred to me (eureka!).

I think it's a latency / rate problem.

In the elapsed time you send an incorrect packet and receive the correction from the server, you will have sent potentially several more incorrect packets, and the server will have to send corrections for them. And if you keep doing that, the errors will never catch up until you send a valid position (i.e. you stop moving).

However, if I get this right, the number of corrections the server should send would basically fit into the latency window. After you receive the first correction, that should re-sync your character to the server (if you are not bumpipng into weird things again) after say, 200ms if you have 200 ms latency.

A couple of things. Play with the tolerance (the thresholds used to detect an out-of-sync packet received from the client), and also, it should be possible to throttle sending corrections to the host (i.e. have a timer so you send corrections only every .1 seconds for example), that could stop the flooding of correction packets.

Thirdly, when using UDP transmissions, inputs and predictions (that happen every 60th second for example) are bundled together on the client and sent at the network rate (say, 20th of a second). They are sent in batch if you will, to avoid multiplying packet. They are also sent duplicated to cover for packet loss.

But using a reliable stream, I dont think that is necessary at all, and you can just write your data straight into the stream.

Another thing you can try for debugging, is to intentionally invalidate your prediction (say, if you press space, add 1 meter in the y position in your client-side calcualtions), and see what happens when the server corrects it. It should ping back in sync pretty quickly. Else, there is a pretty major bug in your code.

What we used to do is to apply one input per frame -> one physics update -> bundle inputs not sent yet -> send bundle at network rate.

We had some problems with making the predictions work EXACTLY the same way on the client and server. Many things came into play and both machines would calculate a slightly different outcome for what we thought was a similar set of inputs.

In the end, the inputs have to be exactly the same, and that means, identifying all your inputs that will affect your character physics.

Everything is better with Metal.

and yeah, as Hellknows says, the right timing is critical. You need to timestamp your inputs accurately, and if you do not use constant frame rates, send the delta time, and you cannot compress the data you send (at least, no quantisation, only lossless compression).

Everything is better with Metal.

Quote:Original post by oliii
something occurred to me (eureka!).

I think it's a latency / rate problem.

In the elapsed time you send an incorrect packet and receive the correction from the server, you will have sent potentially several more incorrect packets, and the server will have to send corrections for them. And if you keep doing that, the errors will never catch up until you send a valid position (i.e. you stop moving).


Exactly. That is the problem that client side rewind & replay fixes.
As soon as a server packet comes in that doesn't match the client position, it rewinds back to the timestamp on the correct packet and replays all the moves it was predicted ahead. Thus, the very next packet that comes in will match the new corrected state, and the dreaded correction loop is entirely avoided.

The bug I have is sort of an edge case to this. It consistently replays all the predicted moves correctly. It's just the very next move, (that hasn't been sent yet) part of which was actually in the prediction/correction cycle, that is incorrect, and starts a new round of replaying.

Quote:
However, if I get this right, the number of corrections the server should send would basically fit into the latency window. After you receive the first correction, that should re-sync your character to the server (if you are not bumpipng into weird things again) after say, 200ms if you have 200 ms latency.

A couple of things. Play with the tolerance (the thresholds used to detect an out-of-sync packet received from the client), and also, it should be possible to throttle sending corrections to the host (i.e. have a timer so you send corrections only every .1 seconds for example), that could stop the flooding of correction packets.


See, the thing is, my server has no concept of the client being out of sync, or having a bad position. It merely takes a set of input, advances the client position, and sends back the results. So it doesn't send any additional corrections when the client messed up. If the client is out of sync, it's always the client's fault, and it's up to the client to fix it.

Quote:
Thirdly, when using UDP transmissions, inputs and predictions (that happen every 60th second for example) are bundled together on the client and sent at the network rate (say, 20th of a second). They are sent in batch if you will, to avoid multiplying packet. They are also sent duplicated to cover for packet loss.

But using a reliable stream, I dont think that is necessary at all, and you can just write your data straight into the stream.


My solution has pretty much the same effect.
Instead of queuing up all the input that happens between network sends, I just poll the input at the same rate as the network so no input piles up on me.

Quote:
Another thing you can try for debugging, is to intentionally invalidate your prediction (say, if you press space, add 1 meter in the y position in your client-side calcualtions), and see what happens when the server corrects it. It should ping back in sync pretty quickly. Else, there is a pretty major bug in your code.


Forcing a change of of the client's position would basically have the same effect as losing a packet. It wouldn't show up for #frames predicted ahead, and it would be effectually the same as having incorrectly predicted ahead, and it would be corrected as such.

Quote:
What we used to do is to apply one input per frame -> one physics update -> bundle inputs not sent yet -> send bundle at network rate.

We had some problems with making the predictions work EXACTLY the same way on the client and server. Many things came into play and both machines would calculate a slightly different outcome for what we thought was a similar set of inputs.

In the end, the inputs have to be exactly the same, and that means, identifying all your inputs that will affect your character physics.


Right now, I'm not doing any prediction on the server. The server just deals with the input as it comes in.
I may need to do some type of prediction down the road, to deal with giving all players a fair playing field regardless of latency. I'm not sure yet, haven't gotten that far. :D


If you gave a helpful reply, I rated you up.
Well.

I restructured it to do one input, physics update and network update per tick.
Still have the same bug.
I did a couple other variations on the same theme, but apparently that's just not where the bug is.

I'm all out of ideas.
If you gave a helpful reply, I rated you up.

This topic is closed to new replies.

Advertisement