Jump to content

  • Log In with Google      Sign In   
  • Create Account


How we solved the infamous sliding bug


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
11 replies to this topic

#1 Oogst   Members   -  Reputation: 349

Like
3Likes
Like

Posted 05 April 2014 - 11:42 AM

I have written a blogpost about the "sliding bug", a bug that has pestered our game Awesomenauts for over 1.5 years. We recently finally fixed it in a patch. I think it is a nice example of the complexities of collision handling in a purely peer-to-peer architecture, so I hope it is okay to make a topic about it here and post a link. An interesting lesson for me was that the solution we used turns out to be really incorrect, but to work really well with actual player behaviour. True correctness is not always needed!

 

How we solved the infamous sliding bug

 

Sliding%20Bug%20-%20Lag%20disagreement.j


My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)


Sponsor:

#2 0BZEN   Crossbones+   -  Reputation: 2010

Like
0Likes
Like

Posted 06 April 2014 - 04:53 PM

It's a very difficult problem, actually, especially peer to peer. Lag is the enemy, even minute. That's why most games run through an authority (server), that controls the game state, while clients merely predict locally what's happening and get corrected by the host. Which causes jumping, re-sync, ect... when two characters collide (see, the original counterstrike). 

 

There are possibilities. Some of them involve using loose tolerances for collisions, making them more soft and forgiving, and letting the collisions resolve themselves on each client. Which can generate rubber-banding physics when you get two players colliding. Players will loose sync as well. What you see on one machine will not match what you see on the other machine, and will take a while to resync. Not nice at all. 

 

Something I always wanted to try, was using in-sync, RTS-style game states. Every player has the same global, deterministic view of the game, with their inputs running at the same ticks, and sequence numbered. Every tick, the game state is saved on each machine, and you sort of 'predict' each remote player's input ahead of time, while you run the local player inputs ahead. Once you receive a remote player inputs, you can rewind the entire game state, and replay the game from that point on to arrive at new locally simulated game state. This means you will have a deterministic game on every machine, and consistent physics across the game. The problem now becomes, what happens when the prediction diverges from the local simulation. And that, I don't know until I can run an experiment myself. 

 

It's a bit how RTS games handle networking. Think of it like Braid, but in a multiplayer context (except Braid of course, breaks the time continuum as part of the game mechanics). 

 

You can also look up Gaffer's networked physics, but in general, that kind of problem you encounter is often resolved ad-hoc, with hacks, basically breaking your internal laws of physics (turning collisions) off for a short while. That's one reason I hate Peer to Peer topology. 

 

It's still valid, we're making games, not shooting rockets at Mars, but it's a bit 'dirty'. Latency is in general, nasty, and a difficult problem. That's why we have all sorts of techniques to hide lag as best we can, like client-prediction, server correction, lag compensation. And even then, it's still in general, crap. Can't bend time, unless you have consistent sub-frame latency, you'll always suffer lag. 


Edited by papalazaru, 06 April 2014 - 04:56 PM.

Everything is better with Metal.


#3 Oogst   Members   -  Reputation: 349

Like
0Likes
Like

Posted 07 April 2014 - 03:38 AM


Something I always wanted to try, was using in-sync, RTS-style game states. Every player has the same global, deterministic view of the game, with their inputs running at the same ticks, and sequence numbered. Every tick, the game state is saved on each machine, and you sort of 'predict' each remote player's input ahead of time, while you run the local player inputs ahead. Once you receive a remote player inputs, you can rewind the entire game state, and replay the game from that point on to arrive at new locally simulated game state. This means you will have a deterministic game on every machine, and consistent physics across the game. The problem now becomes, what happens when the prediction diverges from the local simulation. And that, I don't know until I can run an experiment myself. 

This would be a really interesting experiment! I think it could work, but the big problem is that it is ridiculously complex to build. It is really difficult to write fully deterministic code, so I'd say that the added complexity of building this plus the work in fixing bugs is going to cost an enormous amount of extra development time...


My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)


#4 0BZEN   Crossbones+   -  Reputation: 2010

Like
0Likes
Like

Posted 07 April 2014 - 06:22 AM

Just a pet project of mine :) I wanna see how that would work, if at all. 


Everything is better with Metal.


#5 hplus0603   Moderators   -  Reputation: 5068

Like
0Likes
Like

Posted 07 April 2014 - 03:01 PM

FWIW: That's basically what GGPO does, as well as various other networked physics engines. It's not that hard to build if you can either store a log of physics state for each object, or you can allocate all needed physics state in a single block of RAM. You just have to make sure your engine is set up for it ahead of time, rather than trying to retro-fit it.
enum Bool { True, False, FileNotFound };

#6 0BZEN   Crossbones+   -  Reputation: 2010

Like
0Likes
Like

Posted 07 April 2014 - 03:11 PM

Coo

 

FWIW: That's basically what GGPO does, as well as various other networked physics engines. It's not that hard to build if you can either store a log of physics state for each object, or you can allocate all needed physics state in a single block of RAM. You just have to make sure your engine is set up for it ahead of time, rather than trying to retro-fit it.

 

Cool. Looks like it's worth a look.


Edited by papalazaru, 07 April 2014 - 03:13 PM.

Everything is better with Metal.


#7 Oogst   Members   -  Reputation: 349

Like
0Likes
Like

Posted 08 April 2014 - 01:30 AM

FWIW: That's basically what GGPO does, as well as various other networked physics engines. It's not that hard to build if you can either store a log of physics state for each object, or you can allocate all needed physics state in a single block of RAM. You just have to make sure your engine is set up for it ahead of time, rather than trying to retro-fit it.

Does GGPO also do the full determinism? I know it does the rewind/resimulate thing, but I never heard it also does full determinism.


My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)


#8 Hodgman   Moderators   -  Reputation: 28438

Like
0Likes
Like

Posted 08 April 2014 - 01:43 AM

@Oogst, thanks for sharing the crazy solution biggrin.png

 

@papalazaru - I've seen that basic idea used in 360/PS3 action games.

Each player is constantly trying to commit 'actions' to the game state, with both players having an 'agreed' upon state, and then both players stacking up actions on top of that as 'predictions' locally. Over time, the 'agreed' upon state advances forward by applying the actions of each player. Sometimes this process means that a certain player's predictions have to be undone / popped from their stack, as they're not valid actions any more after the other player's actions have been inserted underneath them.
Sometimes you need a bit of arbitration to resolve conflicts -- e.g. if on the same frame a player carrying a ball says they've kicked it, while on the same frame an enemy says they snatched the ball away. In that case you'd need a rule saying that the person carrying the ball has their actions applied first, etc, etc. Then when trying to apply the "snatched" action, it isn't valid (as the person they're snatching it from doesn't have a ball -- it's now been kicked into the air), so it gets thrown away and the client who performed that action sees a small 'pop' in their gameplay, as the invalid prediction is unwound on their end.


Edited by Hodgman, 08 April 2014 - 01:45 AM.


#9 0BZEN   Crossbones+   -  Reputation: 2010

Like
1Likes
Like

Posted 08 April 2014 - 05:23 AM

 

FWIW: That's basically what GGPO does, as well as various other networked physics engines. It's not that hard to build if you can either store a log of physics state for each object, or you can allocate all needed physics state in a single block of RAM. You just have to make sure your engine is set up for it ahead of time, rather than trying to retro-fit it.

Does GGPO also do the full determinism? I know it does the rewind/resimulate thing, but I never heard it also does full determinism.

 

 

In general, If you think you need prefect determinism, you should be able to adapt your algorithms to use more 'loose' tolerances. 

 

What I'm thinking of :

 

1) some error tolerance on your objects (object position, velocity, ect...), beyond which a de-sync is detected.

 

2) rollback objects affected by the de-sync, re-apply the fix that reduces or eliminate the error.

 

3) replay those de-synced objects up to current time.


Everything is better with Metal.


#10 hplus0603   Moderators   -  Reputation: 5068

Like
0Likes
Like

Posted 08 April 2014 - 10:16 AM

In general, If you think you need prefect determinism, you should be able to adapt your algorithms to use more 'loose' tolerances.


This is the wrong way to think about the problem, in my opinion.

Either you need full determinism, and only send inputs, or you don't need determinism, and you send state checkpoints (perhaps as baselines, with inputs as deltas for some small amount of time.)

Achieving full determinism is absolutely possible, as demonstrated by many games. It's also not easy, as demonstrated by a lot of post mortem write-ups in Game Developer magazine and other resources :-)
enum Bool { True, False, FileNotFound };

#11 SelethD   Members   -  Reputation: 390

Like
0Likes
Like

Posted 08 April 2014 - 11:21 AM

Just my 2 cents, but I would think, unless it really affects gameplay, I wouldnt worry about each screen being a little different than the other.

Not sure exactly what type of game you are making, but I have seen many games where the screens do not match up, but was not necessary for accurate gameplay.



#12 eedok   GDNet+   -  Reputation: 971

Like
0Likes
Like

Posted 08 April 2014 - 03:00 PM

Just my 2 cents, but I would think, unless it really affects gameplay, I wouldnt worry about each screen being a little different than the other.

Not sure exactly what type of game you are making, but I have seen many games where the screens do not match up, but was not necessary for accurate gameplay.

you should try it out, it's a really good game: http://www.awesomenauts.com/

 

on the other hand:

81eDC.jpg
Maybe you shouldn't :P






Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS