Network input handling
Just seeing visual behavior is usually not enough to diagnose things at this level in my experience; it takes analyzing the code and the raw numbers to see what's not lining up.
It may be that I'm just focusing on the wrong details, but it seems to me like the interpolation is at least partially wrong. If interpolation is working correctly, you should smoothly transition between positions even if the positions themselves jitter due to latency correction. As it appears in the stream, your agents are snapping back and forth at regular intervals, which I assume means they're "popping" in sync with tick updates from the network code.
As an experiment, you should have a mechanism which lets you inject artificial ticks of latency into your compensation algorithm. (This is typically as trivial as adding a constant to the expected tick delta between client and server.) Crank this up to some relatively large value, like 1 second worth of ticks. Then move around a bit locally and verify that the interpolation is doing what it should be.
Once you've ruled out interpolation as a problem, you can look back at the latency correction and dejitter buffering to see if there's a bug in there. Generally, being able to artificially expand your latency is very handy for finding those kinds of issues as well.
As an experiment, you should have a mechanism which lets you inject artificial ticks of latency into your compensation algorithm. (This is typically as trivial as adding a constant to the expected tick delta between client and server.) Crank this up to some relatively large value, like 1 second worth of ticks. Then move around a bit locally and verify that the interpolation is doing what it should be.
Once you've ruled out interpolation as a problem, you can look back at the latency correction and dejitter buffering to see if there's a bug in there. Generally, being able to artificially expand your latency is very handy for finding those kinds of issues as well.
Another useful debugging tool is to render shadow copies of objects for different states: Last-received state, current-simulated state, and current-predicted state. Perhaps also "previous tick state" and things like that. Pick a color for each of the different sources of entity position, and have them each be turn-on/off-able through a menu. Render each player more than once, colorized accordingly.
This will let you see where things pop and get an idea of how the different sources of data behave.
This will let you see where things pop and get an idea of how the different sources of data behave.
So, if we wanted to test if the network stuff is the issue, how would we do that? WHat is the best method to see if the problem is the network code or if it is the game logic? We are kinda at a loss.... my only thought was to take the network code and put it into another project but that really seems like a lot of useless work for something we could test in another method. Anyway, any help would be appreciated.
WHat is the best method to see if the problem is the network code or if it is the game logic?
By rendering all three different states for each object. Differences should then become more visible.
Another option is to print out logs of timestamps, entity IDs, and positions sent/received/rendered for each timestep for each host, and compare them for some time window when you think you have a problem. Using Excel or a similar tool is pretty much required for this, as it's a lot of very dense data to sift through.
*nods* Telanor has been dreading doing this, but I am not sure there is any other option at this point! Anyway, thanks, I will get him going on this and see what happens.
Author
This problem really seems to come down to 1 of 2 issues. Occasionally, the client will take a little bit longer on a tick than it should, usually around 40-90ms. I've tried timing various methods and it never seems to come from any specific one. Profiling it is useless since it's a small, infrequent spike. It seems to occur at random.
It wouldn't really be a problem except that after the spike, the client tries to catch up by simulating 2-3 ticks at once, which means the server usually ends up with a couple update packets all at the same time, but it can only process 1 per tick because of the rewinding problem again.
So, the issue is: random spikes throw off the sync (should a game really be expected to run in perfect sync with the server 99.99% of the time, barring network issues?) and the server can't deal with the client playing catchup. I'm really not sure what to do at this point. Are we going to have to just start letting the client tell the server its position and just do some anti-cheat checking on that? Or is there some way to get this to work with this method?
It wouldn't really be a problem except that after the spike, the client tries to catch up by simulating 2-3 ticks at once, which means the server usually ends up with a couple update packets all at the same time, but it can only process 1 per tick because of the rewinding problem again.
So, the issue is: random spikes throw off the sync (should a game really be expected to run in perfect sync with the server 99.99% of the time, barring network issues?) and the server can't deal with the client playing catchup. I'm really not sure what to do at this point. Are we going to have to just start letting the client tell the server its position and just do some anti-cheat checking on that? Or is there some way to get this to work with this method?
Those 90 milliseconds likely come from some background task, or some driver with a poor DPC implementation, or maybe even a garbage collector somewhere if any managed/dynamic runtime is involved.
If you increase the de-jitter buffer size on the server, it will soak up those 90 milliseconds of time. Add 100 milliseconds of de-jitter on receipt, and a 90 millisecond delay won't cause any harm.
But, anyway, if this is rare, then why not just use the correction mechanism to bring the client into order? Assuming you timestamp inputs correctly, a 90 millisecond delay will cause inputs from the client to be deemed "late" and discarded on the server, which means the client and the server will de-sync, so you have to fix the client entity in place on the server until the new, authoritative client state can be sent to the client.
If you increase the de-jitter buffer size on the server, it will soak up those 90 milliseconds of time. Add 100 milliseconds of de-jitter on receipt, and a 90 millisecond delay won't cause any harm.
But, anyway, if this is rare, then why not just use the correction mechanism to bring the client into order? Assuming you timestamp inputs correctly, a 90 millisecond delay will cause inputs from the client to be deemed "late" and discarded on the server, which means the client and the server will de-sync, so you have to fix the client entity in place on the server until the new, authoritative client state can be sent to the client.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
