• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
Heg

Client-Side Prediction & Packet Loss

41 posts in this topic

Hey guys,

I am currently working on my online mode for my Bomberman Clone. I´ve implemented a UDP Layer so far that has some reliability functionality, such as package acknowledgements. I am now implementing the client-side prediction and got a few questions to that.

In all tutorials I´ve read so far, the only things that lets client-side prediction fail are unpredictable events, usually when another player interacts with your player. But what about packet loss? When I move to the right, I can predict the new position pretty easily, but when the command never reaches the server, the new position will be false. Maybe that wouldn´t be such a huge problem, but I´ve got real issues when thinking about planting bombs:

Planting bombs could normally be predicted, since the client knows the rules for planting bombs. You can´t plant bombs when you already planted all of your bombs on the field, or when there is already a bomb at that location. The letter can be a result of the influence of another player, i.e. when someone planted a bomb a tad before you did. This situation would result in the same visual output though, since a bomb is still at the predicted position.

When packets get lost around the bomb planting command, then the prediction would also fail. Maybe the position of the player needed to be corrected, so when he dropped the bomb he actually was at a different location. Another possibility could be that the bomb planted command never reaches the server. I could resend the command, but that would also result in a different bomb location.

Correcting this mistakes could lead to teleporting bombs, which in return could cause trapping yourself. That would be a horrible scenario, since you literally get killed by packet loss. Not predicting bombs on the other hand always results in a bit of unresponsiveness, which is also frustrating. What can I do? Help [img]http://public.gamedev.net//public/style_emoticons/default/biggrin.png[/img] Edited by Heg
0

Share this post


Link to post
Share on other sites
My personal approach here would be to just use TCP, timestamp every command and predict everything the way you do. Even with a retransmitted or slightly delayed command here and there, due to the timestamp you will be able to simulate the game "as if" the command reached the server in time. If state updates from the server reach the client to late, so that the prediction is wrong, well, even AAA titles have this problem (the famous shot around the corner lag problem for example). Guess it's unavoidable with today's networking technology.
2

Share this post


Link to post
Share on other sites
Hi, thanks for your answer.

I don´t see why TCP would be a better option in this case. My UDP Protocol allows me to detect package loss and the order of my packages by using sequence numbers. This leaves me with the option to resend important data or just ignore less important packages, which were recieved either too late or not at all. What I have in mind here is movement data from other players. If those are missing, they don´t need to be resend, because the client doesn´t care about the old data.
1

Share this post


Link to post
Share on other sites
[quote name='Heg' timestamp='1348044395' post='4981592']
but when the command never reaches the server
[/quote]
You can use a hybrid solution. For player commands (relative seldom, low data volume) you can use TCP to take away the burden of self-implemented reliability and use your UDP stack for the rest.

But I'm not sure if a UDP stack is still necessary for smaller projects nowadays, so Heg has a point here.
2

Share this post


Link to post
Share on other sites
you could also use the information from the prediction just to draw the player and wait for the server state packages to draw everything else.
0

Share this post


Link to post
Share on other sites
Hm, [url="http://gafferongames.com/networking-for-game-programmers/"]this tutorial [/url]highly stresses that UDP and TCP shouldn´t be used together, since both are using the IP Protocol and therefore are not independend from one another.

Since I already implemented the UDP functionality I could might aswell use it. I could just resend client input that didn´t go through, including a timestamp, so that the server can reproduce the input. That way the server would have to wait for missing input until it can process the client correctly. I think that wouldn´t be too big of a deal, but I am not sure yet.

What do you mean Inferarum? I think you suggest to just draw player positions and let everything else be approved by the server first. In my case that would result in drawing bombs as soon as the server confirms the position, which would result in a slight delay between pushing the button to plant a bomb and the appearence of the bomb. I fear that this would feel to unresponsive though. What do you guys think?
0

Share this post


Link to post
Share on other sites
[quote name='Ashaman73' timestamp='1348052297' post='4981627']
To be honest, it seems that you build a theoretical [i]horrible [/i]scenario which is really unlikley to happen, this is just a game not a control program for the mars probe.

When the player places a bomb just display it as placement plan, that is, make it translucent and leave it in this state until you receive the ok from the server. The standard case will take a few ms, even with a packetloss, most players will not recognize the delay. And if an other player just placed a bomb on the same location just one single ms before, remove the translucent marker and play some sound to inform the player about the failed operation.
[/quote]

lol :D

Yeah you are propably right. I am really creating the worst case scenario, but someone smarter than me once said "what can go wrong, will go wrong". I guess I follow the advice of you two and wait for the response of the server. I´ll test that with different connection speeds and see how it feels. If it feels too unresponsive, I will add some kind of indicator that the command has been registered by the machine.
0

Share this post


Link to post
Share on other sites
[quote name='Ashaman73' timestamp='1348053217' post='4981634']
[quote name='Heg' timestamp='1348052557' post='4981629']
If it feels too unresponsive, I will add some kind of indicator that the command has been registered by the machine.
[/quote]
You still should send the commands on a reliable channel, your UDP stack fits in here. Just remember the last X send commands + sequence number and the server will send acknowledge messages about the last valid sequence number from time to time (the client can clean up the command queue up to this number). Although the server should send resend-requests when it detects a gap asap.

You can even re-send the same command (unique sequence number) multiple times, using the space left in a UDP for best use and on the server side it doesn't matter at all.
[/quote]

I´ll keep that in mind. I´ll let you guys know how it turns out ;)
0

Share this post


Link to post
Share on other sites
On problem with using TCP and UDP together is that UDP packets tend to be dropped if you start to saturate your bandwidth. If you are nowhere near, then sure. However I prefer a full UDP solution, as it makes things easier in the long run. In any case, you can re-engineer the lower layers if you want to move away from mixing both UDP and TCP.

Secondly TCP can fragment your communication. Say, you send 10 bytes message + 20 bytes message + 10 byte message. Then you may received that data in the form of 12 bytes + 12 bytes + 10 bytes + 6 bytes packets. So you will need some reconstruction at the receiving end. A custom reliable message system piggy-backed on UDP can stop that kind of fragmentation, but really, it's not a big deal to fix that problem. Edited by papalazaru
2

Share this post


Link to post
Share on other sites
[quote name='papalazaru' timestamp='1348093778' post='4981853']
On problem with using TCP and UDP together is that UDP packets tend to be dropped if you start to saturate your bandwidth. If you are nowhere near, then sure. However I prefer a full UDP solution, as it makes things easier in the long run. In any case, you can re-engineer the lower layers if you want to move away from mixing both UDP and TCP.

Secondly TCP can fragment your communication. Say, you send 10 bytes message + 20 bytes message + 10 byte message. Then you may received that data in the form of 12 bytes + 12 bytes + 10 bytes + 6 bytes packets. So you will need some reconstruction at the receiving end. A custom reliable message system piggy-backed on UDP can stop that kind of fragmentation, but really, it's not a big deal to fix that problem.
[/quote]

Yeah, I agree with you. I already programmed a low level UDP layer including an acknowledgement system and all that stuff. So I will definitly use UDP only. I had just some issues around client-side prediction and packet loss. On important game decisions, like planting or exploding bombs, I´ll just wait for a server message and see how that goes. One thing I have to accept is that my game won´t work well when the internet connection is bad. I think there is nothing you can do about that.
0

Share this post


Link to post
Share on other sites
Most games use a varying amounts of smokes and mirrors to mask internet issues.

However, if the connection is bad, it's generally better to let the user suffer for it rather than the game. The game should function well under a minimum requirement of conditions, and anything below that, best forget about it, as long as it doesn't break the game itself.

Planting stuff, or things more akin to 'events' (requesting pick up, climbing into a vehicle, planting a bomb, ...) in time are generally better dealt with with a reliable, but latency-heavy system. That isn't so much of a problem generally, these are often hidden behind client-side animations started at the moment of the request and aknowledged or cancelled by the server response.
1

Share this post


Link to post
Share on other sites
Hm, I could make my guys "bend over" or something in order to plant the bomb :-P A translucent bomb will most likely look odd!
0

Share this post


Link to post
Share on other sites
I´ve got one additional quick question:

Best practice is to send out game updates from the server and input from the client at a fixed rate, say every 100ms. When we define the "effective latency" as the time between issuing a command and the response from the server, this technique does increase the latency by quite a big deal, doesn´t it? For example:

Server: |----------|----------|----------|-----

Client: --|----------|----------|---------|----

The | are the sending points and the time between them is 100ms. If the player on the client issues a command right after input has been send to the server, it would take almost 200ms to get a response, even when the packages arrive instantly. The first X marks the input time, the second one the response from the server:

Server: |----------|----------|----------|-----

Client: --|-X--------|---------X|---------|----

This is noticeable even on a listening server. I fear, when there is actually traveling time needed between the server and the client, that this will have a too big impact. What are your thoughts on that? Is a 100ms interval simply too big? Edited by Heg
0

Share this post


Link to post
Share on other sites
[quote]this technique does increase the latency by quite a big deal, doesn´t it?[/quote]

Yes, it does! This is why CS:S servers running at "tickrate 67" or "tickrate 100" are popular among the low-ping addicts.

On consoles, you typically can't get away with better than 20 times a second (every 50 ms) and many games do it less (every 67 or 100 ms.) Seems to work well for them anyway. The trick is to tune the display and gameplay to work well with whatever your chosen network parameters are.
1

Share this post


Link to post
Share on other sites
[quote name='hplus0603' timestamp='1348337922' post='4982695']
[quote]this technique does increase the latency by quite a big deal, doesn´t it?[/quote]

Yes, it does! This is why CS:S servers running at "tickrate 67" or "tickrate 100" are popular among the low-ping addicts.

On consoles, you typically can't get away with better than 20 times a second (every 50 ms) and many games do it less (every 67 or 100 ms.) Seems to work well for them anyway. The trick is to tune the display and gameplay to work well with whatever your chosen network parameters are.
[/quote]

Okay, so I rely heavily on my client-side prediction, lag compensation and interpolation then. Maybe it´s even a good thing that lag even affects local games. That makes it waaay more easy to test and requires me to work really hard on the networking stuff, otherwise the game will suck for everyone, not just for the guys with slow internet speed. Edited by Heg
0

Share this post


Link to post
Share on other sites
[quote name='Heg' timestamp='1348338528' post='4982697']
Okay, so I rely heavily on my client-side prediction, lag compensation and interpolation then. Maybe it´s even a good thing that lag even affects local games. That makes it waaay more easy to test and requires me to work really hard on the networking stuff, otherwise the game will suck for everyone, not just for the guys with slow internet speed.[/quote]

Yes, you can also generate arbitrary latency for testing how it affects your game.

Usually your game update is staggered. First, Receiving packets, then updating your game (server or client), and finally sending your commands / states at the end of the update. It's one easy way to manage your game loop without adding unnecessary latency.

You can have a virtual network layer that can queue sending and / or receiving packets, as well as simulating packet loss, duplication and out of order (all of which should be invisible outside your lower network layer) for testing purposes, or if you lack the resources for proper internet testing. Then you can use renderless client-bots / servers, or just basic loopback networking with one player, like in Quake3 and many other Unreal / ID / Source games). Edited by papalazaru
1

Share this post


Link to post
Share on other sites
[quote name='papalazaru' timestamp='1348354090' post='4982755']
[quote name='Heg' timestamp='1348338528' post='4982697']
Okay, so I rely heavily on my client-side prediction, lag compensation and interpolation then. Maybe it´s even a good thing that lag even affects local games. That makes it waaay more easy to test and requires me to work really hard on the networking stuff, otherwise the game will suck for everyone, not just for the guys with slow internet speed.[/quote]

Yes, you can also generate arbitrary latency for testing how it affects your game.

Usually your game update is staggered. First, Receiving packets, then updating your game (server or client), and finally sending your commands / states at the end of the update. It's one easy way to manage your game loop without adding unnecessary latency.

You can have a virtual network layer that can queue sending and / or receiving packets, as well as simulating packet loss, duplication and out of order (all of which should be invisible outside your lower network layer) for testing purposes, or if you lack the resources for proper internet testing. Then you can use renderless client-bots / servers, or just basic loopback networking with one player, like in Quake3 and many other Unreal / ID / Source games).
[/quote]

Yeah, I already got a few testing functionality in my network layer, such as latency and packet loss. I am missing functions for duplication and out of order though. I´ll make sure to implement that.

While we are at it, how do you guys handle lost or out of order client input? I would say client input needs to be ordered, but that would also imply that you would have to wait for a resend of lost client packages, since it´s not ordered when you are missing one package.
Should I really wait for such packages, or should I just let the client input get lost? This would result in a correction of the movement on the client, since it predicted the movement wrong.

Current state of the game: Client-Side Prediction and Interpolation seems to work quite well in my current implementation, but I am not reacting to lost, duplicated and out of order packages yet. That´s my next step :-P Edited by Heg
0

Share this post


Link to post
Share on other sites
[quote name='Heg' timestamp='1348487729' post='4983187']
Yeah, I already got a few testing functionality in my network layer, such as latency and packet loss. I am missing functions for duplication and out of order though. I´ll make sure to implement that.

While we are at it, how do you guys handle lost or out of order client input? I would say client input needs to be ordered, but that would also imply that you would have to wait for a resend of lost client packages, since it´s not ordered when you are missing one package.
Should I really wait for such packages, or should I just let the client input get lost? This would result in a correction of the movement on the client, since it predicted the movement wrong.

Current state of the game: Client-Side Prediction and Interpolation seems to work quite well in my current implementation, but I am not reacting to lost, duplicated and out of order packages yet. That´s my next step :-P
[/quote]

I'm toying with sending inputs in a sort of reliable fashion. It's a queue of inputs, and I keep sending them until they are acknowledged. I also have a bandwidth throttle to keep the bandwidth usage in check. If I run out of buffer space (because the server hasn't aknowledged the 3, 4 times the latency or so, or the past second, the buffer with a limited capacity becomes full and I cannot push any more inputs, ect...), then the client would pause, keep resending, until it gets acknowledgements from the server.

It's slightly OTT in a way. Some games (Quake3 for example), naively duplicates inputs (it's a console command you can tweak) so that in case of mild packet loss, you can still reconstruct the stream accurately, with minimum delays. Then if the stream is hopelessly broken (packet loss sometimes happen in bursts, so duplication from one packet to another isn't gonna help you), the server just ignores the client since the break, and correct him to his best ability. It depends how dependent you are on the reliability on your input stream (for example, fully deterministic games like Starcraft 3 which requires fully reliable input streams to work), how much more bandwidth naive duplication would consume, versus how much you have available. Edited by papalazaru
1

Share this post


Link to post
Share on other sites
I'm alse experimenting with client side prediciton / server side correction as follow :


Packets would look like :

client_packet (num_inputs=10) : inputs(sqn=4) + inputs(sqn=5) + inputs(sqn=6) + .... inputs(sqn=13) + predicted_state(sqn=13).

The host checks the predicted_state(sqn=13) by applying client inputs up to sqn=13, and if there is significant discrepencies with the client prediciton, sends a correction.

server_packet : correction_state(sqn=13).

The client can then wind back his player state to (sqn=13), re-apply all the inputs in the queue after (sqn=13), and can carry on sending more prediction to the server, which will hopefully stop deviating after a while.

You could send each prediciton with each input sequence, but that would increase the client packet size significantly. I'm still unsure if you would get benefits from that.
1

Share this post


Link to post
Share on other sites
[quote name='papalazaru' timestamp='1348490188' post='4983199']
I'm toying with sending inputs in a sort of reliable fashion. It's a queue of inputs, and I keep sending them until they are acknowledged. I also have a bandwidth throttle to keep the bandwidth usage in check.[/quote]

This is a fairly standard way of doing it, I've been doing it for a while, there's a gaffer on games article describing this I think:

[url="http://gafferongames.com/"]http://gafferongames.com/[/url]

I think (without looking at the code) I send the server calculated position with the ack for the client input. In fact this may all be part of the regular update packet from server to client (with actor positions).

Then when the client gets the ack, it can compare the server calculated position with its own client side predicted, and if it's different, roll back and recalculate the client position based on the input history. And once it has the server ack up to tick 'blah' it only now needs to send input out from tick 'blah' onwards, etc etc. Edited by lawnjelly
1

Share this post


Link to post
Share on other sites
[quote name='lawnjelly' timestamp='1348493333' post='4983213']
Then when the client gets the ack, it can compare the server calculated position with its own client side predicted, and if it's different, roll back and recalculate the client position based on the input history. And once it has the server ack up to tick 'blah' it only now needs to send input out from tick 'blah' onwards, etc etc.
[/quote]

True. Stamp the server update of that player with the latest input sqn received, send new player position to everyone as part of the regular server update, then the player can use that to check his prediction and correct accordingly. A bit more lightweight, meaning no need for server-side correction, only client-side corrections. Interestingly, it also means that plugging in client prediction into the naive implementation is more straight forward. Edited by papalazaru
1

Share this post


Link to post
Share on other sites
Sending redundant client input is a brilliant idea! I already implemented it. The client now sends input, which is still waiting for server correction, along with new input to the server. I also added some simulation routines to my network layer. A package loss of 10% isn´t even noticeable when looking at the movement of the client. However the opponents do jump occasionally, because of the missing movement information.

The next step would be to implement Lag Compenensation. After that I need some animation to hide the delay between the planting bombs command and the actual appearence of the bomb, and the delay between running over an item and the time when it´s actually collected. I really want to wait for the server response before drawing bombs or collecting items. Edited by Heg
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0