Sign in to follow this  
Webzen

Physics Server / Lag Compensation / Out of sync..

Recommended Posts

Hey everyone,

So I tried to make a picture showing my problem:

LG8g0MT.png

I have a gameserver, but was curious if I need to run a physics engine on the server? I've been following some tutorials online and created some attack animations with a character and would love to create some sort of fighting online side scrolling game.

I know that the client is just supposed to be a dummy sending signals to a host server, however if my host server is not created with my game engine of choice (in this case, Godot's) but (nodejs, c++, elixir, etc), would it still be possible? I just don't know how to handle lag latency compensation. For example, since the character uses set_linear_velocity with a acceleration variable... The amount of time the command actually reaches other players, will have moved the character further than his actual value. That is, if we don't have a "host" managing where everyone really is?

Does this kind of make sense? I'm pretty all over the place. --

I've done simple point and click games with tweens that were multiplayer and it's fairly simple. But when the physics become involved, my brain explodes! Thanks in advance

Edited by Webzen

Share this post


Link to post
Share on other sites

Your problem doesn't actually exist, at least not in the way that you've formulated it. If Player 2 is always 150ms away from the server and Player 1 is always 50ms away, then it takes 200ms (plus some overhead) for the information to travel from Player 2 to Player 1. This total latency is constant and therefore nothing gets out of sync - if Player 2 presses the key, holds it down for 5 seconds, then releases the key, Player 1 will see Player 1 moving left for 5 seconds, just 200ms after it actually happened.

 

Of course, where this falls down is that those figures of 150 and 50 are not actually constant at all. They fluctuate over time depending on network traffic. So Player 2 moving for 5 seconds might appear, to player 1, as 4 seconds, or 6 seconds. It might even appear that way to the server.

 

Generally speaking, you do need a basics physics simulation on the server, or at least "have a "host" managing where everyone really is". You can't just broadcast out state changes for the reason above. I'd love to do my usual thing of recommending the simplest possible approach here, but I'm not confident it would work for a game in a genre that traditionally expects pixel-perfect combat resolution, and where 200ms is an eternity.

Share this post


Link to post
Share on other sites

 

Of course, where this falls down is that those figures of 150 and 50 are not actually constant at all. They fluctuate over time depending on network traffic. So Player 2 moving for 5 seconds might appear, to player 1, as 4 seconds, or 6 seconds. It might even appear that way to the server.

 

Oh lord, this is exactly what I was trying to convey! So glad I am getting help on this, I've always wondered how to solve it.

"have a "host" managing where everyone really is".

Yeah, I think I'm going to do this then. Just run some type of host server to keep track of players locations and whatnot when they give the server a command. I can mess with client prediction stuff to smooth it out, but I need to definitely kink out the host server so it's at least somewhat in sync of positions and whatnot

 

However, it's hard because I'm using Godot as the client side physics engine. And I'm not sure if I can replicate EXACT details of how their physics world is simulated in an actual server network code world. 

Like yeah, I can use a little physics server library netcode now that will work for left,right,top,down,jumping, etc. However the gravity and all that stuff, acceleration, imply_pulse commands and stuff that I use for dynamic combat in Godot is not something I can port over to a network server. Well, I could but I'm not smart enough :P

 

Also, is it possible to just send the state of players to the server? This would help bigly for performance, but result in massive cheating I imagine. But then again.. That doesn't solve the problem you've stated above without a host.

 

I need to stop thinking that I don't need a host.. I know for a fact I need a host. Losing my damn mind  :o 

Edited by Webzen

Share this post


Link to post
Share on other sites

It's possible to get away without a full physics simulation on the server, providing you have enough logic to be able to verify that what you're being sent is legitimate. It's common (although not standard) for clients to send their character's position to the server, and for the server to either accept that data or reject it. This works well for simple position updates (e.g. in an MMO), but significantly less well when trying to do full physics, where it is probably impractical to spot invalid data.

 

Sometimes it's practical to simply run the same engine on the server, but Godot looks to be quite under-developed when it comes to networking, and the only example I could find completely omitted any sort of physics.

Share this post


Link to post
Share on other sites

150 and 50 are not actually constant at all

This is why games introduce additional latency on the receiving side, in a de-jitter buffer. If your jitter is +/- 50 ms, then with a 50 ms de-jitter buffer, you'll end up with latencies of 200 and 100 ms. The increase in latency is a worthwhile price to pay for making sure you have valid data. (And if someone misses that latency window, you can treat that packet as lost.)

 

Separately: You don't HAVE to run physics on the server. You don't even HAVE to have a server. It all depends on what kind of gameplay you're intending to build, how important consistency is, how cheap your simulation is (so how much you can re-play per step) and so forth.

For a few options, look at:

http://gafferongames.com/2004/12/28/zen-of-networked-physics/

http://www.gamasutra.com/view/news/177508/The_lagfighting_techniques_behind_GGPOs_netcode.php

http://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php

They have different approaches, because they build different kinds of games. 

Share this post


Link to post
Share on other sites

Thanks hplus and Kylotan. This forum is pretty damn helpful.. for me just being a new user and you guys responding within minutes.

I've gone ahead with a 10-15 messages per second timer. It basically sends x, y and some other small information to the server. Then, the server broadcasts that data (once its properly checked), to the other players in the map at that same rate.

Then, I use lerping in my fixed_process in Godot and it's actually starting to come along nicely. There is a tiny delay but I can increase the msgs per second for a little bit more reactiveness but it seems fine for now.

Positives of this?

  • No physics server. (A positive for me as I cannot code it!) is what I mean

 

Possible Negatives: 

  • Fly hacking (although, I can try to check the difference between movements on server, but meh)
  • Extremely bandwidth intensive? This, I don't know but I'm thinking I'm going to need to spin up more game instances, which is fine

 

And the server still acts as a host. It will obviously store the players last location in a variable, so when new players join the game they know where they are. 

 

However, this means if the player (client) isn't sending those update packets to the server, sadly they will need to be disconnected. Obviously I'll give a little bit of leeway of the amount as people who lag it might disconnect them falsely

That won't be a problem as anyone who is doing that, is probably fiddling around doing something dirty. Yeah, this is probably not the best way to handle this but with my knowledge the physics server will never get coded. Especially when it has to correspond to Godot's physics.

 

I think I am starting to grasp my head around this though.  It's probably not ideal but for a small indie game I think it'll be fine?

Edited by Webzen

Share this post


Link to post
Share on other sites

The sort of updates you're talking about won't be anywhere near a bandwidth problem for a typical number of players. So don't worry about that.

 

Whether it's sufficient for a small indie game or not is down to other factors. Are you planning on taking people's money for it? If so, you'll want to consider how those people will feel if a cheater ruins their experience. I would suggest implementing at least a few basic checks, such as confirming that maximum velocity isn't exceeded (with some degree of tolerance - again, consider the jitter) and rudimentary collision detection against walls, platforms, etc.

Share this post


Link to post
Share on other sites

Fifteen movement commands per second is fine. I think that's what original Halo used (or perhaps it was 10 ?)

The bandwidth usage is usually not a problem if you use simple packets. Sending x,y,flags as 32-bit integers or floats would be 12 bytes per message, plus some framing overhead (type, length, perhaps client clock information, etc.) You will want to turn on TCP_NODELAY on your socket if you haven't already, to remove the built-in TCP send buffering delay, too.

Generally, you will see more bandwidth being used by the server, as it will send N updates to N players, for N-squared bandwidth usage. If you have 100 players, 16 bytes per player, and 48 bytes of overhead, that's 16*100+48 == 1648 bytes per packet per player times 15 per second equals 1648*100*15 == 2.5 MB/second (25 Mbit/second roughly.)

If you have these 100 players online all the time for a month, that's 6.5 TB. You can easily find hosting that allows you to send that amount of data for under a hundred bucks a month.

If that's too much for you, then the first obvious solution is that getting 100 simultaneous players is actually hard, form a marketing point of view :-) Because bandwidth is N-squared in number of interacting players, 10 simultaneous players will generate 1/100th as much data -- 65 GB/month, or 0.25 Mbit/s. (Well, plus overhead -- 32 KB/sec, or about 81 GB/month, with overhead.)

The second is to reduce the count of players interacting. You may not need to send updates about players that are very far away all that often, or not at all, to other players. (The word for this is "interest management") But you probably have a fair bit to go before that even becomes a problem :-)

Share this post


Link to post
Share on other sites

Wow, you guys are loaded with information. Much more than I can ever attain. Am bookmarking this thread for future references. 

 

Now, I was thinking a bit about this overnight.  What if I made it so I still have a host, items are server-side, cooldowns, etc all the same as before. Except there is a catch. What if player movement was done via P2P? This would drastically remove some overhead, right?

Obviously this would open a yuuge can of worms in terms of fly hacking. But, this would drastically remove the overhead, and make the game seem even smoother than how I am currently interpolating x,y positions. 

However, there has to be a big drawback to this which I'm probably not thinking of...  :ph34r:

Edited by Webzen

Share this post


Link to post
Share on other sites

Four problems with P2P movement:

- The server still needs to know where players are so that it knows who you can target for actions.

- Not all home routers work with P2P gaming setups.

- Every player needs to upload their movements to ALL the other players. This means that every player sends N packets AND receives N packets for each time step. This ends up being more bandwidth use overall, by far, and can easily overwhelm clients with poor upload speeds.

- With P2P systems, players get the IP address of all other players, which leads to bad behavior such as "rage booting."

Bandwidth from server to players is cheap, and the robustness of a central service is typically totally worth it! If you have 100 players online at the same time all the time, chances are you'll be making more than enough money to pay for the server resources :-)

Share this post


Link to post
Share on other sites

..and no P2P movement it is.  :lol:

100 concurrent players online would be amazing. I'm going to use your conservative points about only sending the data when players are in range, if the player is even moving, etc.  So looks like the 15 msgs in/out + host server + interpolated movement  is ideal for me.

Not as reactive as I'd like at 15.. I might up it a bit.. I'll do more testing and might create a new thread later if I got a whole new issue :P 

Share this post


Link to post
Share on other sites

Hey guys. Not meant to hijack this thread. I am interested at the state synchronization being used in Godot engine but I don't understand the code written in Godot. To be honest I don't even know what language it uses. I just want to know how it achieves the state synchronization and I can't find a clear explanation on their gitHub page: https://github.com/empyreanx/godot-state-sync-demo.

 

The few questions in my heads are:

 

1. After I sync the time between server and client, when should I get the first state from the server?

 

2. Since I am doing state synchronization interpolation, I need to have at least two states to interpolate. After the player logs in, when should the player start the interpolation on client side?

 

I have already known the concept of state synchronization but when I roll up my sleeves to work I find I can't get through these questions before moving on.

Edited by caymanbruce

Share this post


Link to post
Share on other sites

Godot apparently uses GDScript, a language only used by the Godot engine: http://docs.godotengine.org/en/stable/reference/gdscript.html

The "state synchronization" demo seems half-baked to me. Specifically, the README seems to indicate that the client and server both run the simulation, and the server sends full snapshots "at a high rate," and presumably the client applies those snapshots when received.

Exactly how it manages the latency of the applied snapshot isn't clear. It says "Linearly interpolated error correction for position and rotation" which I take to mean that it calculates the delta between server snapshot and local state, divides it into some number of frames, and adds that delta to the physics state of the locally simulated object. It says nothing about time delay or rewriting history, so it may not even be able to do this for player characters, or it may introduce a round-trip delay for player entities.

Also, it lists some rather important bits as missing:

  • Jitter buffer
  • Snap state when distance is greater than some threshold

so, I'd assume that this is not the best tutorial on how to do physics state reconciliation between client and server.

Share this post


Link to post
Share on other sites

Godot apparently uses GDScript, a language only used by the Godot engine: http://docs.godotengine.org/en/stable/reference/gdscript.html

The "state synchronization" demo seems half-baked to me. Specifically, the README seems to indicate that the client and server both run the simulation, and the server sends full snapshots "at a high rate," and presumably the client applies those snapshots when received.

Exactly how it manages the latency of the applied snapshot isn't clear. It says "Linearly interpolated error correction for position and rotation" which I take to mean that it calculates the delta between server snapshot and local state, divides it into some number of frames, and adds that delta to the physics state of the locally simulated object. It says nothing about time delay or rewriting history, so it may not even be able to do this for player characters, or it may introduce a round-trip delay for player entities.

Also, it lists some rather important bits as missing:

  • Jitter buffer
  • Snap state when distance is greater than some threshold

so, I'd assume that this is not the best tutorial on how to do physics state reconciliation between client and server.

 

Thank you for pointing that out. Which articles do you recommend if I were to learn to do a proper state synchronization? I know it seems obvious to more experienced developers like you, but I find the information I gathered from the web is fragmented. I just can't put them together to make it work. So far I can do the following:

 

1. syncing the time between clients and the server.

2. Getting latest states from a data structure.

3. lerping between states.

I have also read a bit about lag compensation. But I am stuck in getting start to receive the first state at the correct time and deciding which state to pick due to various latency. 

Share this post


Link to post
Share on other sites

Which articles do you recommend if I were to learn to do a proper state synchronization?

State synchronization is actually a very delicate subject where the specifics of the implementation has significant impact on the specific "feel" of your game. There's many different building blocks, and they go together in a number of ways, depending on your goals.

Good example links used to be listed in the FAQ (top pinned post) of this forum, although some of them have link-rotted.

Good articles I can think of off-hand include:

http://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php

http://www.gamasutra.com/view/news/177508/The_lagfighting_techniques_behind_GGPOs_netcode.php

- http://gafferongames.com/2004/12/28/zen-of-networked-physics/

https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking

Even something as simple as "syncing the time between clients and the server" varies based on what your network model is.

Share this post


Link to post
Share on other sites

 

Which articles do you recommend if I were to learn to do a proper state synchronization?

State synchronization is actually a very delicate subject where the specifics of the implementation has significant impact on the specific "feel" of your game. There's many different building blocks, and they go together in a number of ways, depending on your goals.

Good example links used to be listed in the FAQ (top pinned post) of this forum, although some of them have link-rotted.

Good articles I can think of off-hand include:

http://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php

http://www.gamasutra.com/view/news/177508/The_lagfighting_techniques_behind_GGPOs_netcode.php

- http://gafferongames.com/2004/12/28/zen-of-networked-physics/

https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking

Even something as simple as "syncing the time between clients and the server" varies based on what your network model is.

 

 

Thanks for sharing the references. I have actually read some of them. You are so right about the variation of the "time syncing" part. Yesterday I implemented my first ever state interpolation and if client time and the server time has a time difference less than 10ms the player interpolates very smoothly. However if the time difference is bigger, such as 40ms or 90ms the player will suddenly stop and then after a while just flick to the target position like there is no interpolation at all.

My game demo is very simple. It only shows players moving around on the map. My current interpolation function is like this (written in Javascript):

interpolateState(previousState, currentState, renderTime) {
  if (renderTime <= currentState.timestamp && renderTime >= previousState.timestamp) {
      const total = currentState.timestamp - previousState.timestamp;
      const portion = renderTime - previousState.timestamp;
      const ratio = portion / total;
      player.x = lerp(previousState.x, currentState.x, ratio);
      player.y = lerp(previousState.y, currentState.y, ratio); 
  }
} 

Here server updates every 50ms and the renderTime is the 200ms before the server time which I get from syncing the time with the server at client side. I also put the states I received from the server into a queue which has 3 elements so that on every network update the new state will be pushed into the queue and an existing state will be popped out for use. The queue will only pop a state when it's full, and to interpolate I need to have two states, that's why I choose 200ms before the present time.  This simple and naive approach seems to work but I think there are a lot to improve to make my game playable. Is there anything I can work on to fix the lags due to server and client time difference?

Edited by caymanbruce

Share this post


Link to post
Share on other sites

There's not enough information to go on with what you posted, but here are a few comments:

  • If you're abruptly replacing previousState and currentState when a new value comes in, then obviously the next time you interpolate, the positions will jump.
  • Your interpolation function does nothing if the render time falls outside your 2 timestamps, so that will stop updating the x and y values, causing things to stop. (And then jump to a new position whenever the render time falls within the value range again.)
  • You should keep all states until they're not needed, rather than just 2. Interpolate between the state slightly older than the render time and the state slightly more recent than the render time. Drop states older than the one before the render time, because you'll no longer need them.
  • Hopefully you are updating renderTime smoothly based on the current client, and not having the server change it.

You might consider starting a new thread for your specific problems.

Share this post


Link to post
Share on other sites

There's not enough information to go on with what you posted, but here are a few comments:

  • If you're abruptly replacing previousState and currentState when a new value comes in, then obviously the next time you interpolate, the positions will jump.
  • Your interpolation function does nothing if the render time falls outside your 2 timestamps, so that will stop updating the x and y values, causing things to stop. (And then jump to a new position whenever the render time falls within the value range again.)
  • You should keep all states until they're not needed, rather than just 2. Interpolate between the state slightly older than the render time and the state slightly more recent than the render time. Drop states older than the one before the render time, because you'll no longer need them.
  • Hopefully you are updating renderTime smoothly based on the current client, and not having the server change it.

You might consider starting a new thread for your specific problems.

 

Thanks for your valuable reply. I simply update the client value with the current server value when rendertime is not between the two timestamps. Sorry I haven't posted all my code because the whole thing will make it harder to read. As per your suggestion I rephrase my question and put it in a new thread: https://www.gamedev.net/topic/687361-slow-response-in-state-interpolation-when-time-gap-is-big-between-client-and-server/

 

And what do you mean by 

  • Hopefully you are updating renderTime smoothly based on the current client, and not having the server change it.

?

I get the server time and then rollback 200ms as the render time. I am using a third party module call timesync (https://github.com/enmasseio/timesync) to sync the time between server and client and it seems to be doing the trick.  If you don't mind you can reply in the other thread I posted so I will stop here.

Edited by caymanbruce

Share this post


Link to post
Share on other sites

What if you don't get a new server time when you need to render the next image?

Typically, you should keep an offset between "local clock time" and "server clock time."

Then, when time comes to render, you read the local clock time, and add the offset, and render the game at that point in time. This will often imply doing interpolation or extrapolation.

Note that "time" may be measured in ticks or seconds, and "render" may also include intermediate simulation step updates. It may even be that your network rate runs at one interval (say, 50 milliseconds,) your simulation runs at another interval (say, 8 milliseconds,) and your render runs at a third interval (say, 16.7 milliseconds,) and you have to use interpolation to go from network to simulation, and then again to go from simulation to render.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this