Websocket-based realtime multiplayer game client and server communication

Started by
7 comments, last by caymanbruce 7 years, 2 months ago

I am building my client/server realtime MMO game using websocket which is based on TCP. My game will be server authoritative and will use non-lockstep. That means all game logic (not graphical) will run on the server and players can join and leave the game any time they want.

Although I have read the Valve Network model and Quake 3's snapshot architecture and all the entity interpolation stuff on http://gabrielgambetta.com/, I am not very clear about the actual communication between the server and the clients because the models are specifically targeted against UDP transport.

One question is that do I still need to synchronize the ticks on clients and the server if I use websocket? So far as I know, TCP guarantees that the packets are sent orderly and if part of the packets are lost they will be resent. With this in mind, the state (of other players) on the client should always be one tick before the state the server is ready to send out to the client. Please correct me if I am wrong. And the server will probably need to maintain a queue to store all the input commands the client just sent out. On the next tick of the server, the server executes the input commands and send out new states to the client.

So I imagine the process should look like this:

  1. client send input commands to the server =>
  2. server receives commands and queue the input commands =>
  3. on next tick server update the game state using all the input commands in the queue and send the game state out =>
  4. client receives new game state and interpolate other players on client from current state to the new state.

I am not sure if this is the correct way to do it. How about the edge case? When the game starts new player joins the game after the server has run certain amount of ticks, I guess I wouldn't need to worry about the tick number? Hope someone can give me some guidance here.

Advertisement

This is a big topic.

In general, whether you use TCP or UDP doesn't change the networking implementation all that much.

UDP may sometimes lose or delay packets.

TCP may sometimes delay packets, although it tends to do so more often than UDP (which instead loses or re-orders the packets)

In order to make sure that you don't build up too much of a backlog on a stalled TCP connection, it's a good idea to keep a measure of how many packets you've sent since you last received a packet from the other end, and if you find youself sending, say, more than three packets without hearing anything, stop sending packets until you get a packet from the other side.

In general, each side will have a timer, and it will decide, X times per second, to formulate and send a packet to the other side. That packet will include whatever information is necessary to make the game synchronization work the way you want it. If you stop sending packets because of a TCP lag event, then you should take this into account -- it will be similar to how UDP compensates for dropped packets.

enum Bool { True, False, FileNotFound };

How about the edge case? When the game starts new player joins the game after the server has run certain amount of ticks, I guess I wouldn't need to worry about the tick number?

The question is, what are you using the tick number for? Often, FPS logic is overkill for MMOs and you can just render the entities at whatever position the server says they are, at the time you get the message. Given that you want some interpolation to smooth things out you can usually say that whatever update you receive right now is the exact state you want to render in N milliseconds time (where N might be something between 50 and 500, I'd say) and you just interpolate from the previous position (or, the currently interpolated position) towards that.

How about the edge case? When the game starts new player joins the game after the server has run certain amount of ticks, I guess I wouldn't need to worry about the tick number?

The question is, what are you using the tick number for? Often, FPS logic is overkill for MMOs and you can just render the entities at whatever position the server says they are, at the time you get the message. Given that you want some interpolation to smooth things out you can usually say that whatever update you receive right now is the exact state you want to render in N milliseconds time (where N might be something between 50 and 500, I'd say) and you just interpolate from the previous position (or, the currently interpolated position) towards that.

Sorry I am new to writing MMO games. The reason I use tick is that I have chatted with someone who shows his demo of the exact same type of game on Youtube and he gave me his example code which uses snapshot interpolation and each snapshot has a timestamp. But I find that timestamp is not easy to synchronize between client and server so I googled again and someone said using the server tick is enough for the purpose of interpolation.

At anytime the client receives the state, the interpolation code looks like this:


var total = oldState.timestamp - newState.timestamp
var portion = renderTime - oldState.timestamp
var ratio = portion / total
// interpolation stuff...

If I use tick number the code may be a little different but I hope you get the idea. However I am still a bit confused even for now. If I can just interpolate using the current state and the incoming state without worrying about the timer, how do I decide how much time to interpolate given that the interval between each state received may vary? Can I just use the interval of the state update loop on the server? For example, 20 times a second for sending out new states on the server which does a 60 FPS physics loop.

This is a big topic.

In general, whether you use TCP or UDP doesn't change the networking implementation all that much.

UDP may sometimes lose or delay packets.

TCP may sometimes delay packets, although it tends to do so more often than UDP (which instead loses or re-orders the packets)

In order to make sure that you don't build up too much of a backlog on a stalled TCP connection, it's a good idea to keep a measure of how many packets you've sent since you last received a packet from the other end, and if you find youself sending, say, more than three packets without hearing anything, stop sending packets until you get a packet from the other side.

In general, each side will have a timer, and it will decide, X times per second, to formulate and send a packet to the other side. That packet will include whatever information is necessary to make the game synchronization work the way you want it. If you stop sending packets because of a TCP lag event, then you should take this into account -- it will be similar to how UDP compensates for dropped packets.

What you said makes me feel it is a very big topic. I haven't thought about throttling packets sending before. I am very new to making MMO. I hope there is a simpler version so I can set it up quicker without killing my brain.

Sorry I am new to writing MMO games. The reason I use tick is that I have chatted with someone who shows his demo of the exact same type of game on Youtube and he gave me his example code which uses snapshot interpolation and each snapshot has a timestamp. But I find that timestamp is not easy to synchronize between client and server so I googled again and someone said using the server tick is enough for the purpose of interpolation.

At anytime the client receives the state, the interpolation code looks like this:


var total = oldState.timestamp - newState.timestamp
var portion = renderTime - oldState.timestamp
var ratio = portion / total
// interpolation stuff...
If I use tick number the code may be a little different but I hope you get the idea. However I am still a bit confused even for now. If I can just interpolate using the current state and the incoming state without worrying about the timer, how do I decide how much time to interpolate given that the interval between each state received may vary? Can I just use the interval of the state update loop on the server? For example, 20 times a second for sending out new states on the server which does a 60 FPS physics loop.

It's not clear what your code snippet means since you talk about 'ticks' but don't reference them at all in that code. Interpolating via a ratio is fine and uncontroversial - what is interesting is how you define the 2 values to interpolate between, and therefore how you calculate the ratio.

Ultimately there are about 10 different permutations of ways to slice and dice this problem, most of which depend on the exact problem you're trying to solve. But if you want to use something like the Valve system - overkill for RPGs but effective for shooters - then you probably want to do something like the following:

  • broadly synchronise your client timer to the server timer by measuring latency before the game starts;
  • accept the timestamp on each message you receive;
  • render the worldstate X milliseconds in the past (relative to the current timer), interpolating between 2 past states, where X is a number like 100 (canonical Valve example, although old - you can probably reduce this)
  • consider resynchronising the timers as the game goes on in case latency drifts

What I was suggesting is something simpler which is usually fine for games where precise synchronisation is not an issue (perhaps because the game is relatively slow-paced or there are no aiming mechanics)

  • when a message is received, timestamp it with the client's local time plus some arbitrary delay value (again, 100ms is reasonable) to make it a 'future' state
  • render the worldstate 'now', interpolating between the worldstates either side of that time

The latter can have some disadvantages - for instance network jitter can make the character appear to speed up or slow down - but there are ways to ameliorate that (e.g. fudge the 100 value up or down as latency changes, or add a second layer of interpolation) and sometimes you just don't see it as a problem in practice, depending on how your simulation works.

Sorry I am new to writing MMO games. The reason I use tick is that I have chatted with someone who shows his demo of the exact same type of game on Youtube and he gave me his example code which uses snapshot interpolation and each snapshot has a timestamp. But I find that timestamp is not easy to synchronize between client and server so I googled again and someone said using the server tick is enough for the purpose of interpolation.

At anytime the client receives the state, the interpolation code looks like this:


var total = oldState.timestamp - newState.timestamp
var portion = renderTime - oldState.timestamp
var ratio = portion / total
// interpolation stuff...
If I use tick number the code may be a little different but I hope you get the idea. However I am still a bit confused even for now. If I can just interpolate using the current state and the incoming state without worrying about the timer, how do I decide how much time to interpolate given that the interval between each state received may vary? Can I just use the interval of the state update loop on the server? For example, 20 times a second for sending out new states on the server which does a 60 FPS physics loop.

It's not clear what your code snippet means since you talk about 'ticks' but don't reference them at all in that code. Interpolating via a ratio is fine and uncontroversial - what is interesting is how you define the 2 values to interpolate between, and therefore how you calculate the ratio.

Ultimately there are about 10 different permutations of ways to slice and dice this problem, most of which depend on the exact problem you're trying to solve. But if you want to use something like the Valve system - overkill for RPGs but effective for shooters - then you probably want to do something like the following:

  • broadly synchronise your client timer to the server timer by measuring latency before the game starts;
  • accept the timestamp on each message you receive;
  • render the worldstate X milliseconds in the past (relative to the current timer), interpolating between 2 past states, where X is a number like 100 (canonical Valve example, although old - you can probably reduce this)
  • consider resynchronising the timers as the game goes on in case latency drifts

What I was suggesting is something simpler which is usually fine for games where precise synchronisation is not an issue (perhaps because the game is relatively slow-paced or there are no aiming mechanics)

  • when a message is received, timestamp it with the client's local time plus some arbitrary delay value (again, 100ms is reasonable) to make it a 'future' state
  • render the worldstate 'now', interpolating between the worldstates either side of that time

The latter can have some disadvantages - for instance network jitter can make the character appear to speed up or slow down - but there are ways to ameliorate that (e.g. fudge the 100 value up or down as latency changes, or add a second layer of interpolation) and sometimes you just don't see it as a problem in practice, depending on how your simulation works.

This is the snippet I received from someone who uses timestamp and I am going to revise it to use tick because I don't know how to synchronize the timer between server and client and I am not sure if using a timestamp is necessary here. And I haven't implemented the ticks in the code because I have the question at the top of this topic.

Thanks for pointing out there are many ways to solve the problem. I am building a realtime MMO, not like FPS but more like agar.io so it is not a slow-paced game in my opinion and a lot of data could be transported every second if many players are on the screen.

A tick basically is a timestamp, synchronised between server and client - it's just at a lower level of granularity. It's like a clock with the minute hand removed, so all you see is the current hour. You can't get around the need to synchronise, but with a tick you're usually saying that it's not as closely synced.

Going back to your case where a new player joins the server, what they need to know depends on how you run your game. If you want to synchronise based on clocks, they need to deduce the server clock, or an approximation thereof, in order to know how to handle incoming messages. If you're using ticks, it's exactly the same, except the client needs to know the current tick. (Again, ticks are just a low-resolution clock. Look at the Valve doc to see this in the illustrations.) If you're using neither, like my simplified approach, you just assume the message is for 'now' and pretend there's no server latency to worry about.

Looking at open source implementations of agar.io, I'm seeing no evidence that they attempt any type of latency compensation at all. Most likely they just lerp towards whatever the most recent state is, over a predetermined short time span. Note that this might not be the Valve/Quake approach of "render an interpolated position between 2 states", more like "interpolate the actual state based on snapshots, and render whatever that is".

A tick basically is a timestamp, synchronised between server and client - it's just at a lower level of granularity. It's like a clock with the minute hand removed, so all you see is the current hour. You can't get around the need to synchronise, but with a tick you're usually saying that it's not as closely synced.

Going back to your case where a new player joins the server, what they need to know depends on how you run your game. If you want to synchronise based on clocks, they need to deduce the server clock, or an approximation thereof, in order to know how to handle incoming messages. If you're using ticks, it's exactly the same, except the client needs to know the current tick. (Again, ticks are just a low-resolution clock. Look at the Valve doc to see this in the illustrations.) If you're using neither, like my simplified approach, you just assume the message is for 'now' and pretend there's no server latency to worry about.

Looking at open source implementations of agar.io, I'm seeing no evidence that they attempt any type of latency compensation at all. Most likely they just lerp towards whatever the most recent state is, over a predetermined short time span. Note that this might not be the Valve/Quake approach of "render an interpolated position between 2 states", more like "interpolate the actual state based on snapshots, and render whatever that is".

The open source implementation of agar.io is only for learning purpose. I don't think it is ready for production for thousands of players in different locations. It lacks a lot of implementation of networking stuff. When I don't use timer at all I can see the jittering clearly so I will definitely try something better to reduce latency.

This topic is closed to new replies.

Advertisement