Clientside prediction / Rewind + Replay bug

Started by
22 comments, last by hplus0603 15 years, 11 months ago
When out of ideas, add logging!

On the client, for each timestep, log the user input, the simulation state, the step number, and what you're sending to the server.
For each received packet from the server, log the intended timestep and the server state.

On the server, log each time step, the state for each player for that timestep, what state you're sending to each player, and what input (for which timestep) you receive from the client.

Now, go through the logs, and compare the client and server states and inputs for each timestep. Processing your log into an Excel file might help with this, as will formatting the log to make it easy to machine parse. After doing this, you ought to figure out what the pre-/postconditions are for the bug firing, which in turn will let you insert conditional breakpoints, or asserts, to start debugging the problem.

Oh, and it's often useful to use a virtual clock, which always advances X milliseconds per loop of the game. That will let you stop and debug without losing the next time step, which you would if you used a real clock. Similarly, on the server, advance the clock to the earliest clock received from a client, but no more, which will let you halt the server when halting a client. And, finally, for debugging a multi-client situation, you should not let the clients advance more than what the server tells them they can, so that breaking one client in the debugger will effectively stop all the clients and the server, until that client comes back.

Turn all that off for release, though, or crashing players (or cheaters) will cause problems.


Good luck!
enum Bool { True, False, FileNotFound };
Advertisement
Quote:Original post by hplus0603
When out of ideas, add logging!

On the client, for each timestep, log the user input, the simulation state, the step number, and what you're sending to the server.
For each received packet from the server, log the intended timestep and the server state.

On the server, log each time step, the state for each player for that timestep, what state you're sending to each player, and what input (for which timestep) you receive from the client.

Now, go through the logs, and compare the client and server states and inputs for each timestep. Processing your log into an Excel file might help with this, as will formatting the log to make it easy to machine parse. After doing this, you ought to figure out what the pre-/postconditions are for the bug firing, which in turn will let you insert conditional breakpoints, or asserts, to start debugging the problem.


Belive me, I've done all that. Repeatedly. In depth. Ad infinitum.
Everything matches up perfectly... Except that sometimes, even though everything syncs up, the numbers are wrong.
If you gave a helpful reply, I rated you up.
Hmm, here's another thing you can try.

What does your movement code look like so far? Does it have things like velocity?

If so, try to simplify it and make it just so that the player is moved by a fixed amount whenever he has a directional key down. Otherwise, he stays in place.

Something like...
// Input command processingif (command.getKey() == UP_ARROW) player.MoveForward(5);else if (command.getKey() == DOWN_ARROW) player.MoveForward(-5);else if (command.getKey() == LEFT_ARROW) player.MoveSide(5);else if (command.getKey() == RIGHT_ARROW) player.MoveSide(-5);else { /* Do nothing, player position is unchanged */ }
So that the arrows keys control your player's velocity directly, rather than indirectly through acceleration.

This way, assuming no input commands are dropped from the client, the player local prediction and server authoritative updates should match without a problem.

If there is one input command packet dropped, it should take only one update from the server to 'fix' it on the client (i.e. snap his position to the right place) and you should not be seeing your 'feedback loop of corrections' happening (unless the bug is still elsewhere).

If your controls actually control the player's acceleration only, then when the server sends out an update containing a corrected position but not velocity, there will be more than 1 corrections occuring until you let go of the key (since the velocity on the client will be greater than on the server).

I'm not sure how this applies to your problem, but maybe it'll help. Unfortunately, I wasn't able to try out your code because I don't have C# installed at home atm.
If you have full data logging, and you have full input logging, then you should have a reproducible case, where running the same program with the same input (from the log) each time will generate the same bug. You should be able to then single-step the program at the right time step, to figure out what is going wrong.

If you can't make input logging/playback work consistently, then that should tell you something about the bug, too :-)
enum Bool { True, False, FileNotFound };

This topic is closed to new replies.

Advertisement