We have been investigating wireless latency (WiFi and cellular) and its effect on realtime multiplayer games.

First, we have some questions about real world wireless latency. It seems that limited latency testing in the wild is not sufficient for preparing a game for real world use. Ookla speedtest is the most used network testing tool with 5 million tests a day, but we found that it only gives the best out of 10 samples, so this naturally gives highly optimistic results. What are better sources of real world latency data?

Looking at latency data, what’s a good way to decide the range of latency a game should accommodate? For example, you can look at the min/max/standard deviation, the mean, or the tail (95th percentile).

What is the best practice to actually test the effect of latency on a game? Is it better to test with deterministic or random latencies? For example, for deterministic testing you might use a constant latency at 95th percentile. Deterministic latencies make it easier to find issues in a controlled and repeatable manner, while random latencies are a closer to what your game will see out in the real world.

We have posted more details about this topic and are very interested in what other developers have to say on the matter. If you are interested in reading more about our thought process, check out our post about mobile latency here.

I have a strange habit of assuming nearly-worst-case scenario with end-user situations. Hardware, internet, etc... and aim to make everybody as happy as possible, without disappointing those who are more invested in their setups.

Thanks for your responses, guys.

I'm not aware of any good, thorough latency data. Your best bet is probably to try a number of different devices in different service areas. For example, a home 100 Mbps connection in Korea, a 4G MiFi WiFi adapter in the US, a 20 Mbps DSL connection in Europe, and a 1 Mbps satellite-downlink modem-uplink connection may all perform very differently along both loss, latency, jitter, and bandwidth.

If you're designing your game to be played well, then you should design for at least 99th percentile, if not 99.9th or better. Although percentiles are also quite hard to pin down -- latency and jitter may vary more over macro time (time of day, day of week, internet weather) than it varies from packet to packet.

hplus, how does one go about field testing on all the different connections you mentioned? Would one use a third party testing company for field testing? It seems expensive/time-consuming for a typical studio to do this kind of testing and make it statistically meaningful. It appears hard to test and troubleshoot issues with latency using just field tests during development.

Our opinions below are informed by experience in other fields. We are hoping to understand current best practices in game development.

A little more background information about our perspective— personal computing has been moving from hard-wired to wirelessly connected devices. Portable devices (smart phones, tablets and laptops) use only wireless connections, and even consoles and PC’s increasingly use wireless connections. Realtime multiplayer action games work best on hard-wired connections. This transition from hard-wired to wireless connections require developers to address higher and more variable latency.

We’d like to start off by making a distinction between field tests in the wild and lab tests during development. During development, there is a need for regular regression testing of builds; this would include unit, integrated functional and performance tests. Regression tests are most useful when automated (run without human input).

Lab tests, both automated and live-player, require setting up repeatable, yet representative network conditions. This allows direct comparison of test results to assess software quality and uncover bugs. In contrast, for comparing field tests in the wild, both network conditions and quality of the software are variable.

How does one know where to test, and under what conditions? And if one has covered the instances that the customers are going to see? If we spend most of our time testing for the tails of the distribution, we may not have confidence in the outcomes majority of players will experience. The further you go out into testing the tail (i.e. a higher percentile) in field tests, the more tests you have to run for the data to be statistically significant (adding to the cost). Field tests are more expensive than lab tests, so it is better to use fewer field tests and more extensive lab tests.

Network engineers have been concerned with similar issues for a long time. IETF RFC 3393 addresses measurement of packet delay (latency) variations for IP Performance Metrics. Interestingly, it deprecates use of the term “jitter” due to ambiguity. It views latency as a distribution whose statistics are inferred from measurements. Network emulators are commonly used in development of networking products. Statistical parameters of the distribution are then used to configure the emulated test conditions. In that case, testing confidence is improved by long enough tests and importance sampling (stress testing). Repeatability in statistical tests can be achieved by using pseudo-random sequences.

Thanks again for your insight into the industry.

Latency on WiFi connections is actually just as high and variable as on cellular networks. This is due to the quality of the radio channel (fading) and the traffic, and has little to do with the hardware involved. Recent data collected over a 6-month period by OpenSignal demonstrates this point and is summarized here: http://opensignal.com/blog/2014/03/10/lte-latency-how-does-it-compare-to-other-technologies/

OpenSignal collects data from over 1.2 billion WiFi points and 800,000 cell towers.

Our limited testing confirms this as well: http://sugarcanegames.com/latencysummary2.html