124 replies to this topic

[Kylotan]

Is it just me, or is basically everybody asking exactly the same question here these days? It's a fundamental rule of physics that you if you want perfect information you have to wait for it, or you can have predicted information but at some point it will inevitably need correcting. Perfect synchronisation across 2 computers is impossible where human input is involved; all you get to do is move the slider between "more latency" at one end and "more errors" on the other, and no amount of trying to reconstruct the 'real' situation by resimulating inputs and trying to guess when they actually happened will fix it. Once you accept that, your network programming life becomes simpler.

[Telanor]

It's not that I'm trying to make things perfect. If you saw how the game is acting currently, you'd understand. It's simply unacceptable, no one would want to play with this amount of jittering on a local connection. Every other game runs nearly flawlessly under ideal network conditions, I don't see why ours shouldn't.

HPlus: About the dejitter buffer, I gave it a try before and it seemed to make it a bit worse, but it's probably because I screwed something up. I'll look at it some more and see if I can find what I did wrong.

[hplus0603]

It's a fundamental rule of physics that you if you want perfect information you have to wait for it, or you can have predicted information but at some point it will inevitably need correcting.

No, it's not a rule of physics. It's a law of signal transmission theory.

[riuthamus]

It's a fundamental rule of physics that you if you want perfect information you have to wait for it, or you can have predicted information but at some point it will inevitably need correcting.

No, it's not a rule of physics. It's a law of signal transmission theory.

 

I know many of you have been helping to get us this far, but we are near a point of hitting a wall. Would any of you be willing to take a look and maybe guide us in the right direction? Seeing the project for yourself would very much improve your understanding of what we are doing and maybe help us realize where we are flawed. I am not asking for code to be done but really for somebody with the knowledge to slap us on the face and say... silly go here!

 

If compensation is required you can speak with me and we can make that happen. Again, i appreciate all efforts thus far. If there is anything I can do to help this place grow please let me know

[ApochPiQ]

From my experience, when you're stuck on a nasty problem like this, the best thing to do is divide and conquer.

Inevitably bugs like this crop up when you have half a dozen minor things all wrong. Fixing one of them may actually make the others seem worse, so it's easy to run around in circles for weeks without making any real visible progress.

What you need to do at this point is break things down as far as you can. I don't know your code obviously, but here's how I'd go about getting the problems fixed:

• Define your simulation model on paper - how it should work ideally
• Nail down all your theoretical edge cases and stuff in your head first, so you know what the target looks like
• Reduce this to a set of implementation steps
• Break down each step into the logic components needed to accomplish it
• Break it down further
• No, really, you're still thinking too big.
• Once you have a list of extremely tiny (think 2-3 line of code) operations, it's time to debug them
• Unit test everything you possibly can
• Collect shitloads of data on the behavior of everything you can't effectively unit test
• Go back and unit test a bunch of that stuff anyways because you really can and it just seems ugly
• Your guiding principle at this point should be that you can prove that, in isolation, any one of your moving parts is doing its job correctly. The up-front theory work is mandatory, because it does no good to combine a bunch of working parts into a whole that does the wrong job.

You should set up your code and development tools so that you can poke your network model on any number of axes independently. Want some latency? Crank up a slider. Packet loss? 'Nother slider. Want to make sure you interpolate between simulation frames and render frames accurately? FPS regulation slider. Need to know if your desync checks are working? Add fake noise to the client to see when the server detects a desync and forces a resync.

And so on.


This can easily seem insurmountable but it's just a matter of disciplined engineering to get it done. In the end it's better for you guys as developers to tackle this yourself instead of shelling out to another programmer to do the grunt work. You already know your goals better than any outsider, and you know what your expectations are. Just establishing that would run you up several grand in consulting fees if you brought in someone new.

Learn to love Excel and dozens of megabytes of log data; and remember, break it down into tiny parts. You'll be fine :-)

Edited by ApochPiQ, 07 January 2013 - 06:00 AM.

[hplus0603]

You still haven't answered whether you are able to pin an entity in place on the server for the time it takes to re-sync the entity state on the client.

[Telanor]

No we haven't done that yet. How would the server even know there has been a prediction mismatch until after the problem has been solved?

[ApochPiQ]

Typically, you compare the authoritative position/velocity data with the reported data coming from the client. Anything over a tiny threshold == desync, at which point you start throwing away reports until the problem is rectified by retransmitting authoritative state from the server. This includes both position discrepancies and time discrepancies so you can effectively lock your agents in place during packet-loss situations and other transient network issues.

[hplus0603]

No we haven't done that yet. How would the server even know there has been a prediction mismatch until after the problem has been solved?

 

The server can notice that it didn't get input from the client for a particular step, for example.

 

Or the client could send a checksum or replica of its important state (like position) on some (or all) time steps, and the server will compare to what it gets itself.

[riuthamus]

Well, it looks like we fixed the prediction part. We have to do some more extensive testing but we certainly have seen an improvement.  We are working on fixing some some of the camera issues now.. that are associated with this but otherwise.... we are on track right now. Ill let you know or telanor will if there is more information that shows up.

[Kylotan]

It's a fundamental rule of physics that you if you want perfect information you have to wait for it, or you can have predicted information but at some point it will inevitably need correcting.

No, it's not a rule of physics. It's a law of signal transmission theory.

 

The last time I checked, signal transmissions were governed by the laws of physics. Signals travel over a medium at the speed of light or below which means you always receive a signal some period of time after it was sent, and during that period the recipient cannot possibly know what has changed at the sending end.

 

If quantum computing becomes common - not just for the internet but within each computer - maybe it will become possible for 2 computers to be in perfect sync. Until then, physics prevents it because signals can't be sent faster than the speed of light.

[Kylotan]

It's not that I'm trying to make things perfect. If you saw how the game is acting currently, you'd understand. It's simply unacceptable, no one would want to play with this amount of jittering on a local connection. Every other game runs nearly flawlessly under ideal network conditions, I don't see why ours shouldn't.

 

Sorry, I'm a bit snappy because this is the 3rd thread I've seen on this forum in recent days which basically has exactly the same problem, ie. people are trying to synchronise the client to the server and having an awful time of it, probably because they're trying too hard to get things 'in sync' using complex systems like the Valve networking model that was designed for fast-paced shooters. Keep it simple. When I write networked games I don't even try to have all the computers in sync. Clients always have their character slightly ahead and other characters slightly behind, and it doesn't matter unless your game is so fast paced that the distance a character moves in 100ms is going to make or break the gameplay.

[Telanor]

So things are working quite well now, thanks for all your help so far guys. The dejitter buffer solved the issue of the spikes, along with another change I made to allow the client a chance to catch up before forcing a resync.

I'm starting to work on getting other aspects synced and I was wondering what kind of things should use client-side prediction and what shouldn't. For example, player skills/attacks and world modification (placing and destroying blocks).

Also, what about things that involve random elements? Such as AI or maybe a cannon that has a slightly random firing angle. Does that require waiting on a server response?

[ApochPiQ]

Assuming by "prediction" what you mean is extrapolating from previously known state: the rule of thumb is to predict anything you can. Failing that, predict anything that is sensitive to latency. As a layer of filtering, don't predict anything that costs more to undo than it costs to wait for the authoritative update. If you can predict world geometry changes well enough, do it; but if you hit a point where it takes, say, 60ms to undo the modification once you get updated world state, it's probably time to just eat the latency and let the server control it completely. This is kind of an art form and is going to vary heavily per game.

Randomness is a tricky one. If you're aiming for full determinism, just share a PRNG seed and the current state of the generator when clients enter the game. If you have any doubts whatsoever about being able to maintain a fully deterministic simulation (hint: if you're not doing lockstep and/or you do any kind of on-the-fly state correction, you probably aren't fully deterministic) then just let the server be authoritative.

There is a hybrid approach, of course. If you know that some action will result in, say, 20 consecutive PRNG rolls, you can avoid transmitting all 20 roll values from the server by just transmitting a random seed to the client and having the client generate the 20 numbers itself. This is basically just a form of data compression.

[Telanor]

Ok so I guess since we're not doing a fully deterministic simulation, we'll have to wait on the server for random based stuff, but the rest of it should be predicted.

[riuthamus]

Are there any real bonuses to doing full determinism over what we have now?

[ApochPiQ]

Determinism is nice for some things (replays, validation, etc.) but it's an extremely difficult thing to do if you haven't designed for it from day one.

[Kylotan]

In many cases you don't need to predict anything, but can hide the latency anyway. eg. Start a player attack animation as soon as the player clicks on the target, but only show the damage information when the client receives it from the server.

 

In my opinion the only cases where you really need to try running simulations in parallel are at 2 ends of the gaming spectrum - twitch games like shooters where every millisecond counts, and strategy games where it's imperative that everybody has the same data but there is too much data to send. Beyond that, a simple interpolation layer on the movement is usually enough.

[Telanor]

I'm having a problem figuring out how to deal with syncing creation of entities. Suppose you have an important entity like an arrow where it wouldn't be acceptable to wait for the server response before you fire the arrow. How can the client predict this locally? At first I thought, oh, just create a temporary entity and when the server acknowledges it, use the ID the server assigned so it can be synced. But then how do you match up that entity with the one the server will create when it processes the player's "fire" command?

[ApochPiQ]

Why is it unacceptable? You either have an authoritative server on arrow-firing, or you don't.