Jump to content

  • Log In with Google      Sign In   
  • Create Account

Can bandwidth variation cause extra lag/loss?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
14 replies to this topic

#1 Oogst   Members   -  Reputation: 393

Like
1Likes
Like

Posted 04 August 2014 - 04:04 AM

In Awesomenauts we recently reduced both bandwidth and packet count a LOT, but it seems like this is actually creating more issues for a number of issues. A theory we have on why this is, is that some internet connections perform better under constant high load than under varying lower load. Is this true?

 

Before the patch each client used to send 150 packets per second all the time, using around 20kB/s upload. This is for a six player match.

 

After the improvements we now send between 50 and 120 packets per second and use between 8 and 13kB/s.

 

So bandwidth and packet count have greatly been reduced under ALL circumstances, but have become a lot less constant. Might this variation in itself be a cause for new problems?


My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)


Sponsor:

#2 kunos   Crossbones+   -  Reputation: 2207

Like
0Likes
Like

Posted 04 August 2014 - 04:12 AM

I also see an increase of packet loss when I reduce the size of the packets, but not the rate. It seems that some routers prefer bigger packets.


Stefano Casillo
Lead Programmer
TWITTER: @KunosStefano
AssettoCorsa - netKar PRO - Kunos Simulazioni

#3 samoth   Crossbones+   -  Reputation: 4783

Like
3Likes
Like

Posted 04 August 2014 - 05:13 AM

It may help to elaborate what "more issues" you are seeing. If that is the fact that packet loss becomes more painful, then that's only natural. If you have fewer packets sent less often (with much less redundancy), then every single packet is crucial. Losing just one is much more noticeable than it would be otherwise (when you'd maybe lose 3 or 4 but wouldn't care!).

 

As for packet sizes and frequencies, the size of each packet should actually not matter on the internet (as long as you don't exceed the MTU) since routers work on a packet-per-second notion and have number-of-packet length queues. They do not really care about how much is in one packet, and due to minimum frame lengths and padding, sending a small packet and a large packet is more or less the same on the physical cable, too.

That isn't true on ATM obviously (last mile for DSL home users and much of the "mobile" stuff), since an ATM frame has an entire 48 bytes of payload. So yeah, quite clearly smaller packets cause fewer ATM frames (which should actually be better!). It also matters on wireless, since other than in a cable, bit errors due to noise do actually happen. On a properly shielded ethernet cable, the chance for that to happen is around 10-13 to 10-14, which comes down to zero for most practical purposes, but surely that's not the case for Wifi when your neighbour turns on the hoover...

 

Also, the frequency of packets may very well matter. While common logic tells you "fewer is better", this may not always be true.

 

A lot of wireless networks (including the one I use here, and I don't even know how you could turn this off!) throttle down when the device thinks that there isn't enough traffic. Not sure how it works exactly, but devices on the 2.4GHz net often run with 20-28 Mbit/s if you don't touch them or don't do a lot and as soon as you do something serious switch to 54MBit/s. Similarly, my laptop is currently shown between 292 and 299 MBit/s (that's on 5GHz), which is the "usual" high-power value that I see whenever I bother to look, and 100MBit/s being the low-power one. The box has "600" written on it, so I guess it would probably go twice as fast, if the laptop's network card allowed for it.

Assuming it works similarly for other people, then quite possibly, you may get double (or triple?) bandwidth for some users when you send more packets.

 

Also, there is a statistical thing to it which might affect you in presence of congestion. Routers have relatively short forward queues and discard quickly. This is done on purpose nowadays. When RAM became cheaper, the initial assumption was that one could improve routers by putting in more RAM and making queues longer. This proved wrong because if a router keeps a packet for too long before forwarding it, it will be considered "lost" and will already be resent, ending up as duplicate packet and wasting bandwidth.

Common logic tells you that if the router is already congested, you should send less. However, routers discard quickly, and not necessarily in-order. So it is possible that sending more packets on a congested router gives you a higher likelihood of getting some through. Though of course the opposite may as well be the case due to bandwidth quotas / QoS / DoS prevention, and whatnot. Impossible to tell, really.

 

In summary, packets can be lost and will be lost. You just have to be able to cope with that fact, there's not much you can do about it.



#4 Oogst   Members   -  Reputation: 393

Like
0Likes
Like

Posted 04 August 2014 - 07:09 AM

Hmm, so our hypothesis that sending less might create new problems is actually true then... Argh! How would you suggest we handle this? Basically the only thing I can reasonably tweak is sending more deliberately when it is not really needed. But how can the game know when it needs to do that?

 

An idea behind this patch was that if the bandwidth is usually lower, then the connection is not congested yet when a short peak happens, making it better able to handle shorter periods of more bandwidth. Is this idea not true at all then, or only not true for some users?

 

As for the type of network issues: we have seen several. Some users claim higher ping overall. Others claim to have more network errors (connection lost entirely). Others claim that during longer teamfights there is first more lag and then it settles and play fine again. I think only the latter would fit exactly what you describe: the connection doesn't immediately scale up when the game starts sending more, but if it keeps using more than it does scale up after a while.

 

We did think of the situation where packet loss becomes more relevant and were already wondering whether we might want to send more stuff twice now even before the time to wait for an acknowledgement has passed.

 

(And thanks for the extensive reply, highly appreciated! worshippy.gif)


Edited by Oogst, 04 August 2014 - 07:10 AM.

My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)


#5 hplus0603   Moderators   -  Reputation: 5309

Like
2Likes
Like

Posted 04 August 2014 - 09:04 AM

our hypothesis that sending less might create new problems is actually true then


In general, you should see lower server load, and more players able to play well, when using less data and fewer packets.
The one area where fewer/less may hurt you is if you previously had redundancy, and now you don't, and you see occasional-but-not-infrequent packet loss.
In general, most well-designed router and transmission systems I know of would perform equally-or-better with fewer/smaller packets.
However, there may be all kinds of ill-conceived devices in the way between a player and the packet destination, not least of which is the player's wifi router or internet gateway. That may be a 10 year old device that thought that prioritizing larger/bigger streams would lead to better throughput benchmark numbers -- who knows?
If you can quantify what the packets are, and how you use them, and how much "loss" or other "degradation" you are seeing, it would be easier to make a better judgment on what's going on.

Btw: 50 to 120 packets per second is still a whole lot. I presume that you send packet-per-command instead of bundling multiple messages into a single packet and sending on a fixed schedule?
enum Bool { True, False, FileNotFound };

#6 KnolanCross   Members   -  Reputation: 1293

Like
0Likes
Like

Posted 04 August 2014 - 09:12 AM

Hmm, so our hypothesis that sending less might create new problems is actually true then... Argh! How would you suggest we handle this? Basically the only thing I can reasonably tweak is sending more deliberately when it is not really needed. But how can the game know when it needs to do that?

 

An idea behind this patch was that if the bandwidth is usually lower, then the connection is not congested yet when a short peak happens, making it better able to handle shorter periods of more bandwidth. Is this idea not true at all then, or only not true for some users?

 

As for the type of network issues: we have seen several. Some users claim higher ping overall. Others claim to have more network errors (connection lost entirely). Others claim that during longer teamfights there is first more lag and then it settles and play fine again. I think only the latter would fit exactly what you describe: the connection doesn't immediately scale up when the game starts sending more, but if it keeps using more than it does scale up after a while.

 

We did think of the situation where packet loss becomes more relevant and were already wondering whether we might want to send more stuff twice now even before the time to wait for an acknowledgement has passed.

 

(And thanks for the extensive reply, highly appreciated! worshippy.gif)

 

Options that I can think of, to solve the problem in a short time:

1) Create a NOP-like package, just to create some padding. Use some in-client statistics to define what is the ideal padding (remember to limit the max padding, in order not to be ddosed).

2) Support both protocols (the current one, and the former with bigger packages). Let players change it themselves, or create an statics on latency, use one protocol sometimes, then the other. Select the one that performs better.

 

A long time solution would probably be trying to define if some specific hardware works better with the former protocol and create a more use friendly option on the ui, for instance a checkbox saying: "Use old format network (works better with older modems/devices/routers*)"

Finally, never discard the possibility of a bug in either the client or the server code.

 

* I don't know what would be the best way to phrase it, but assume users do not know the technichal language.


Currently working on a scene editor for ORX (http://orx-project.org), using kivy (http://kivy.org).


#7 samoth   Crossbones+   -  Reputation: 4783

Like
0Likes
Like

Posted 04 August 2014 - 09:15 AM

Btw: 50 to 120 packets per second is still a whole lot.

It would be interesting to know if this is for 6 players (that's what I assumed!), or for one. If that's per player, it is indeed a lot.

 

Hmm, so our hypothesis that sending less might create new problems is actually true then... Argh! How would you suggest we handle this?

For some people, yes. But probably not for all. I would not try to handle this at all. You cannot ever make something that works for everybody. This is simply impossible.

 

Do send what you must send (but not more), and make it work for 95% of your users 95% of the time. That's as good as you can get. Yes, some people will still complain, but you can't make it work for everyone.

Do not "economize" and try to be super-smart to squeeze out a few extra bytes, but then don't send what you must send, or make your protocol fragile if a single datagram doesn't arrive as you expected. That's saving on the wrong end.

 

Do assume that packet loss will happen, because it just will (occasionally, but unavoidably). You don't own or control the internet, so there's not much you can do either. Do however not assume that it happens all the time and in a large, measurable quantity because it doesn't (well, it does, this still happens, but only when you either got congestion control totally wrong, or when something is broken that's beyond your powers to fix!).


Edited by samoth, 04 August 2014 - 09:24 AM.


#8 Oogst   Members   -  Reputation: 393

Like
0Likes
Like

Posted 04 August 2014 - 10:08 AM

The 50 to 120 packets per second is indeed a lot. It is what one player sends to all five players combined. So when it is 50, each player sends 10 packets to each other player. Awesomenauts is entirely peer to peer, so each player needs to inform every other player of his own status. This number is for upload only: each player also receives a total of 50 to 120 packets per second.

 

We do need to handle things like this: we are seeing way too many network problems in Awesomenauts at the moment and way too high lag so we need to look into every weird little detail to find where we can improve something. We received dozens of reports from players who claim they got more problems since the network optimisation, so we cannot discard that. Especially since Awesomenauts is competitive multiplayer, so if one of the six players has a problem, everyone in the match is affected.


My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)


#9 hplus0603   Moderators   -  Reputation: 5309

Like
0Likes
Like

Posted 04 August 2014 - 04:19 PM

Thanks for clarifying. 50-100 packets per second when sending to five other players is not unreasonable, other than peer-to-peer being generally unreasonable as a design choice :-) (I think that, for you, that ship already sailed, so no more about that.)

Lag is not a well-defined word in the greater world, unfortunately. When your players say "lag," you should make very clear about what that actually means. I've heard users call "lag" on anything -- for some users, it's bad frame rate. For other users, it's occasional network corrections. For yet other users, it's how fast the character accelerates. For other users, it's how many frames the graphics card keeps buffering, and yet others will say that it's when they see one thing but another player sees another thing. Lag can be caused by thermal management in laptops, or fluorescent lights being on in the room, or a room-mate watching the latest episode of House of Cards on Netflix. If all they're complaining about is "lag," you don't know what they're complaining about! In that case, you should dive into a lot more detail, including visiting players in the same city as you to see what they're experiencing first hand.

If you can't visit any users at all (sadness!) then at least you may be able to collect statistics on ping times, packet loss, packet send/receive rate, frame rate, CPU utilization, and perhaps system temperature, and upload this kind of data to your home servers every few minutes from randomly selected users. This will give you some more hard data to correlate to. You might even have a dialog after a match where users get to rate "My opponents were nice/bad," "I had fun/it sucked," and "The game was laggy/okay," and upload the statistics together with the survey results. You can't fix what you can't quantify and measure!

Edited by hplus0603, 04 August 2014 - 04:20 PM.

enum Bool { True, False, FileNotFound };

#10 Oogst   Members   -  Reputation: 393

Like
0Likes
Like

Posted 05 August 2014 - 01:54 AM

Yeah, it is often difficult to interpret what users mean by "lag". We recently had a user who complained about lag and it turned out that the cause was that sounds were played 50ms too late, causing the game to feel less responsive...

 

We have been monitoring this for ages and have tons of network metrics, so we do know very well what issues users are having. The main issues we see are:

-Jitter and packet loss causing character movement to not look fluent

-High latency causing situations similar to the infamous shooting-around-the-corner problem

-Connection loss causing players to be disconnected

-Latency spikes / short periods of no packets coming through

 

We have lots of users and we know that each of these is happening too much, so we are looking for anything we can find to improve on any of these. We had hoped that general bandwidth/packet count decrease would do a lot for all of those. It did in a sense: the number of network errors was reduces by around 15% I think since this patch.


My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)


#11 hplus0603   Moderators   -  Reputation: 5309

Like
1Likes
Like

Posted 05 August 2014 - 09:47 AM

-Jitter and packet loss causing character movement to not look fluent
-High latency causing situations similar to the infamous shooting-around-the-corner problem
-Connection loss causing players to be disconnected
-Latency spikes / short periods of no packets coming through


In some sense, "welcome to the internet." :-(

For the last week, I've had extreme connection problems between my and the Guild Wars 2 servers, with frequent disconnects and client crashes. This is a major title, with pretty reasonably skilled operators, using controlled servers, connecting from a major metro area (Silicon Valley.) Once you introduce user hosts (many of them, for peer-to-peer,) then the problem just gets worse.

The fact that you think it's a problem that high latency causes the shot-around-the-corner problem means that you are thinking about the design in terms of local LAN multiplayer. Unfortunately, the internet is nothing like that. You may, very well, not be able to achieve what you want to achieve through technical means. You seem to be looking for "some way to make the network less terrible," rather than looking for "some way to make the game and game design more robust in the face of a terrible network." I totally understand the urge, but I'm not optimistic you will make great strides in that direction.

If I had to look in that direction, I would attempt to set the QoS field of the packets, and perhaps switch to a UDP port that's frequently used for other real-time needs (such as IP telephony or IPTV,) and see if some networks pay attention to these things. Chances are, most users would be unaffected, and perhaps some home gateways would barf on the QoS bits.

The real solution to the problem of the internet is more likely to include some combination of:
- visualization of detected quality and problems to users in real time
- education of users for what they can do, and what they should expect
- change the game design to be more tolerant of latency (yes, this makes some particular game designs impossible!)
- change the technical implementation to be more tolerant of changing network conditions (adaptive de-jitter buffer, smoothing in position display, tolerant of temporary black holes, etc)
enum Bool { True, False, FileNotFound };

#12 starbasecitadel   Members   -  Reputation: 699

Like
0Likes
Like

Posted 05 August 2014 - 10:57 AM

Just checked out your game's home page, it looks awesome! (pun intended) :)    

 

Is it pure UDP or do parts of your game such as lobby use TCP?   If it is pure UDP it might help me with a side networking project I'm developing.

 

In general, I tend to agree with hplus, where you as much as possible should assume the internet is unreliable.  Doing so usually means a slightly worse game experience for the players who have great connectivity but a far more playable experience for the substantial chunk of players who have packet loss > 0.25% etc. 



#13 Oogst   Members   -  Reputation: 393

Like
0Likes
Like

Posted 05 August 2014 - 12:59 PM

You are absolutely right that the internet always has tons of problems. However, with good code that CAN be improved upon. Perfection is impossible, but there are various degrees of imperfection. Through the improvements in our latest two patches we managed to reduce disconnects in Awesomenauts by 25%. (These patches improved throttling and reduced bandwidth/packet count.) I am sure there is still much that we can win and that is where I am looking right now.

 

So even though you are right that part of the solution to networking issues is in game design and education, I am at the moment much more interested in the actual network improvements we can still do.

 

Based on your replies one thing we are going to try is to send certain important data twice, not waiting for the acknowledgement, so that a packet loss will be much less problematic.

 

You mentioned visualising connection quality to users. We already show pings, what else could we do for this? Awesomenauts uses automated matchmaking, so users cannot select matches themselves from a lobby with ping information, if that's what you meant.

 

Just checked out your game's home page, it looks awesome! (pun intended) smile.png    

 

Is it pure UDP or do parts of your game such as lobby use TCP?   If it is pure UDP it might help me with a side networking project I'm developing.

Everything we do is indeed pure UDP, but I don't know what is used for matchmaking: that goes through Steam's systems and is a black box to us. So matchmaking might be TCP. (For the moment: we have begun working on our own matchmaking systems to have more control over that.)

 

We do our own reliability because packet count is so important to us, Awesomenauts being peer-to-peer. Within one of our packets all kinds of data is combined, some of which is sent reliably (and thus resent if no acknowledgement comes in), while other parts of it are completely unreliable.


My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)


#14 hplus0603   Moderators   -  Reputation: 5309

Like
0Likes
Like

Posted 05 August 2014 - 06:49 PM

with good code that CAN be improved upon


Agreed! That's the "adopting the game implementation to the network" strategy, rather than the "make the network itself better" strategy :-)
 

So matchmaking might be TCP. (For the moment: we have begun working on our own matchmaking systems to have more control over that.)


Are you actually having trouble with the matchmaking system? Also, if they do NAT introduction for you, then they have to use UDP if you use UDP if you want the introduction to work.
Also, you should be able to easily find out what they do, using Ethereal or similar.
 

You mentioned visualising connection quality to users. We already show pings, what else could we do for this?


As an example: Let's assume you display each other connected user's user name in the top-left in a list.
In that list, you could also display the ping time to that user (which it sounds like you do,) as well as an estimate of network quality for that user (which each user can generate and update the other users with,) as well as a small graph of network quality that scrolls perhaps 40 pixels wide; low/white meaning good, high/red meaning packet loss and high ping times.
The network quality indicator I would calculate would be something like the ping time to each other user, plus the amount of packet loss to each other user, scored using some weighting system.

Now, a user that cares about networking can look at the status in the list, and tell whether it's just him/her, or if everyone is seeing problems. When something weird happens, the user can look at the list in the top-left, and if one user has a high/red line scrolling by, the problem is with that user; if all users have the same high/red lines, it means it's the local connection that's the problem. Over time, users may then learn that, when that happens, they need to yell at dad to pause downloading German porn or whatever else may be getting in the way.

This is just an example; I don't know what would work best for your situation and your users, but something that lets a user tell "it's me" versus "it's him" and something that lets a user tell "the last 10 seconds had X quality to user Y" will likely let users self-diagnose better.


But, who knows, there may be low hanging fruit inside your own code, too. That's much harder for me to say anything about, because I don't work intimately with that code! I can just suggest making sure you have full-network-stream recording available (and the playback that goes with that,) as well as a deep knowledge of Ethereal or tcpdump.

Edited by hplus0603, 05 August 2014 - 06:50 PM.

enum Bool { True, False, FileNotFound };

#15 Oogst   Members   -  Reputation: 393

Like
0Likes
Like

Posted 06 August 2014 - 02:36 AM


Are you actually having trouble with the matchmaking system?

Yes, but that is on a wholly different side of things: we want to use different kinds of matchmaking algorithms than Steam makes possible with their system. And Steam goes down too often, kicking players from matches because we rely on Steam too much. It is somewhat related though: we think we can do better at selecting hosts and matching players based on connection quality if we have more control over the matchmaking algorithms.


Edited by Oogst, 06 August 2014 - 02:36 AM.

My dev blog
Ronimo Games (my game dev company)
Awesomenauts (2D MOBA for Steam/PS4/PS3/360)
Swords & Soldiers (2D RTS for Wii/PS3/Steam/mobile)

Swords & Soldiers 2 (coming to WiiU)
Proun (abstract racing game for PC, coming to mobile)
Cello Fortress (live performance game controlled by cello)





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS