-Jitter and packet loss causing character movement to not look fluent
-High latency causing situations similar to the infamous shooting-around-the-corner problem
-Connection loss causing players to be disconnected
-Latency spikes / short periods of no packets coming through

In some sense, "welcome to the internet." :-(

For the last week, I've had extreme connection problems between my and the Guild Wars 2 servers, with frequent disconnects and client crashes. This is a major title, with pretty reasonably skilled operators, using controlled servers, connecting from a major metro area (Silicon Valley.) Once you introduce user hosts (many of them, for peer-to-peer,) then the problem just gets worse.

The fact that you think it's a problem that high latency causes the shot-around-the-corner problem means that you are thinking about the design in terms of local LAN multiplayer. Unfortunately, the internet is nothing like that. You may, very well, not be able to achieve what you want to achieve through technical means. You seem to be looking for "some way to make the network less terrible," rather than looking for "some way to make the game and game design more robust in the face of a terrible network." I totally understand the urge, but I'm not optimistic you will make great strides in that direction.

If I had to look in that direction, I would attempt to set the QoS field of the packets, and perhaps switch to a UDP port that's frequently used for other real-time needs (such as IP telephony or IPTV,) and see if some networks pay attention to these things. Chances are, most users would be unaffected, and perhaps some home gateways would barf on the QoS bits.

The real solution to the problem of the internet is more likely to include some combination of:
- visualization of detected quality and problems to users in real time
- education of users for what they can do, and what they should expect
- change the game design to be more tolerant of latency (yes, this makes some particular game designs impossible!)
- change the technical implementation to be more tolerant of changing network conditions (adaptive de-jitter buffer, smoothing in position display, tolerant of temporary black holes, etc)

You are absolutely right that the internet always has tons of problems. However, with good code that CAN be improved upon. Perfection is impossible, but there are various degrees of imperfection. Through the improvements in our latest two patches we managed to reduce disconnects in Awesomenauts by 25%. (These patches improved throttling and reduced bandwidth/packet count.) I am sure there is still much that we can win and that is where I am looking right now.

So even though you are right that part of the solution to networking issues is in game design and education, I am at the moment much more interested in the actual network improvements we can still do.

Based on your replies one thing we are going to try is to send certain important data twice, not waiting for the acknowledgement, so that a packet loss will be much less problematic.

You mentioned visualising connection quality to users. We already show pings, what else could we do for this? Awesomenauts uses automated matchmaking, so users cannot select matches themselves from a lobby with ping information, if that's what you meant.

Just checked out your game's home page, it looks awesome! (pun intended)

Is it pure UDP or do parts of your game such as lobby use TCP? If it is pure UDP it might help me with a side networking project I'm developing.

Everything we do is indeed pure UDP, but I don't know what is used for matchmaking: that goes through Steam's systems and is a black box to us. So matchmaking might be TCP. (For the moment: we have begun working on our own matchmaking systems to have more control over that.)

We do our own reliability because packet count is so important to us, Awesomenauts being peer-to-peer. Within one of our packets all kinds of data is combined, some of which is sent reliably (and thus resent if no acknowledgement comes in), while other parts of it are completely unreliable.

with good code that CAN be improved upon

Agreed! That's the "adopting the game implementation to the network" strategy, rather than the "make the network itself better" strategy :-)

So matchmaking might be TCP. (For the moment: we have begun working on our own matchmaking systems to have more control over that.)

Are you actually having trouble with the matchmaking system? Also, if they do NAT introduction for you, then they have to use UDP if you use UDP if you want the introduction to work.
Also, you should be able to easily find out what they do, using Ethereal or similar.

You mentioned visualising connection quality to users. We already show pings, what else could we do for this?

As an example: Let's assume you display each other connected user's user name in the top-left in a list.
In that list, you could also display the ping time to that user (which it sounds like you do,) as well as an estimate of network quality for that user (which each user can generate and update the other users with,) as well as a small graph of network quality that scrolls perhaps 40 pixels wide; low/white meaning good, high/red meaning packet loss and high ping times.
The network quality indicator I would calculate would be something like the ping time to each other user, plus the amount of packet loss to each other user, scored using some weighting system.

Now, a user that cares about networking can look at the status in the list, and tell whether it's just him/her, or if everyone is seeing problems. When something weird happens, the user can look at the list in the top-left, and if one user has a high/red line scrolling by, the problem is with that user; if all users have the same high/red lines, it means it's the local connection that's the problem. Over time, users may then learn that, when that happens, they need to yell at dad to pause downloading German porn or whatever else may be getting in the way.

This is just an example; I don't know what would work best for your situation and your users, but something that lets a user tell "it's me" versus "it's him" and something that lets a user tell "the last 10 seconds had X quality to user Y" will likely let users self-diagnose better.


But, who knows, there may be low hanging fruit inside your own code, too. That's much harder for me to say anything about, because I don't work intimately with that code! I can just suggest making sure you have full-network-stream recording available (and the playback that goes with that,) as well as a deep knowledge of Ethereal or tcpdump.

Are you actually having trouble with the matchmaking system?

Yes, but that is on a wholly different side of things: we want to use different kinds of matchmaking algorithms than Steam makes possible with their system. And Steam goes down too often, kicking players from matches because we rely on Steam too much. It is somewhat related though: we think we can do better at selecting hosts and matching players based on connection quality if we have more control over the matchmaking algorithms.