I forget which version the "third" version is, but is that perhaps the version that requires a full round-trip to the server before you see the movement? Or is it the version where actors aren't allowed to interact/collide with each other directly? In either of those cases, you won't run into cases where the server simulates differently from you, other than if your simulation itself is nondeterministic. In those cases, you have the option of backing up and re-simulating each time a snapshot is received, or applying additional "drift" to the simulation model to compensate for whatever differences you detect between local and server states.

Can you link to which of gaffer's work you're following?

On 1/25/2019 at 2:34 PM, Bow_vernon said:

However, nowhere in his materials did he say that there is some kind of physics rewinding. I am trying to follow his third method, "state synchronization". I wonder what did he do when the client receive server's snapshots (consists of states + input, not just state like in interpolation method)? He didn't mention to rewind the physics to the server's snapshot and replaying all inputs up to current client's tick.

I'm not sure if he does this (I'll have to re-read his work ) but that's exactly what we do when using synchronized physics simulation in our racing game. It does add a lot of extra CPU load on the clients, especially at high ping...
In the "Counter Strike" model, the reverse happens -- the server pays the cost of constantly rewinding / fast-forwarding.

Also, just for clarity, the clients actually have to simulate / extrapolate to a point in time that's at least ping*0.5ms ahead of the server clock -- so that their input commands will arrive at the server just before the point in time when they should be applied to the simulation. We do a clock-synchronization process during connection to get a common wall clock for each client and the server to refer to, and then the clients can adjust their in-game clock appropriately to maintain themselves in the future slightly.

To support massive player counts and/or high ping servers, we also support the snapshot interpolation method as a server configuration setting.

The Jitter Buffer section implies that state updates from the server arrive at the clients slightly before the client is due to present that frame. This means that he's running the server in the future / clients in the past (like counter strike) and not how I've described in my last post. 

The downside here is, in order for Sever updates packets for frame F to arrive just before frame F occurs for a client, the client clock has to be synced to at least ping/2 ms in the past relative to the server clock. This means that when the client sends user input to the server for frame G, the server will receive it at time G+ping. This will be fine for completely predictable actions, like moving a single FPS character -- there is latency, but the client can hide it by applying the inputs on their end at frame G -- this is incorrect / divergent from the server's version of events (because the server will apply the change at G+ping), but if no other interactions occur, the end result / final position of the character will be the same on both ends.  You only run into issues when your predictions are wrong because there's other things going on in the world, e.g. When two different FPS characters walk into each other, or you walk into a door/moving object being controlled on the server, or if two people throw physics objects at each other, there's some kind of timing/rhythm puzzle, etc, you'll notice ping ms of input latency. AFAIK, counter strike doesn't address that at all / it's just a tolerated downside. However, they do address the special case of weapon ray-tracing, by having the server rewind all player skeletons when receiving a shooting input, so that input lag of shooting is corrected for. The mostly correct predictions plus rewinding of weapon hit boxes means that it seems rock solid. 

21 hours ago, Bow_vernon said:

It does add CPU Time though, but if it's the correct way of doing things then I'm okay. Kinda weird though since it feels like we're punishing the lagging client lol.

So far the client usually have to rewind 5-6 ticks, and capped at 16 ticks at max, which would make it jittery if there's high ping in the client.

I don't know if it's the "correct" way to do things, I'm just making up game/engine architecture as I go along.  I'll let you know after I successfully launch a game using this model

For what it's worth though, Doom 3 used this model of client-side rewind and reprediction. There's a very in depth write up here: http://mrelusive.com/publications/papers/The-DOOM-III-Network-Architecture.pdf

As for the punishing ping thing... Yeah that's my main objection to this model at the moment. I'd love to support fairly high pings, as my racing game doesn't have collisions, so high ping shouldn't be as much of an issue. 

I haven't implemented these yet, but my plans are to automatically tweak some settings on the client when their ping is too high:

* increase the physics tick size during reprediction - e.g. Instead of doing 10x 60Hz ticks, run 5x 30Hz ticks to repredict. 

* spread reprediction cost over multiple frames. e.g. When the client gets a state update that's 10 frames old, they currently save their game state, load the servers state snapshot, run 10 ticks, then compare the new state against their saved one and add differences into the visual smoothing buffer. Instead, when getting that 10 frame old server snapshot, then could save their game state,load the servers state snapshot, run 5 ticks, save this as a new fake "server snapshot", load own their saved game state from before this prediction and advance to the next frame as if nothing happened. Next frame they take that half repredicted gamestate from the previous frame (which is now only 6 ticks out of date instead of 10) and pretend that it just arrived from the server. 

Quote

increase the physics tick size during reprediction

This will guarantee that your physics simulations do not match up between client and server.

Unreal does something like this (frame rate dependent simulation) and thus doesn't have 1:1 positions on client/server.

The basic idea behind "zen of networked character physics" is to keep them together, though.

In general, though, assuming you don't resimulate your entire physics world, but only the player, simulating 10 steps should be very cheap. You may only need to do this if the difference at that past step between your old snapshot and the received checkpoint is greater than some small delta, although you make take the GGPO approach and just always re-simulate to show the player "as correct as possible" -- if it's cheap, might as well! Computers are fast.

3 hours ago, hplus0603 said:

This will guarantee that your physics simulations do not match up between client and server.

They already don't match up before this  (we use PhysX which is non-deterministic, and also sometimes user inputs from clients will arrive slightly too late to be applied at the exact frame that they wanted to apply them on) so the visual position of each player's vehicle is smoothed out to hide the corrections, while the physics state is constantly snapping with corrections from the server.

If I end up using a larger tick size for high ping players, they'll end up with more physics snapping = more weird visual smoothing... but the severity of these issues will depend on how bad their ping is. I think gamers kind of expect to have a worse network experience in fast paced when their ping is high, so it might be acceptable..?
Using the client-side reprediction model, the current behavior is that high-ping players end up with higher and higher CPU loads  / their framerate drops lower as their ping increases!! That's pretty awful, so I think having horrible physics quality at high ping is a better outcome than horrible framerate