The final result is a toolkit that can be exploited to experiment and tune different game communication scenarios. The full source code of the demo is freely available on GitHub. In this article, I will take you through the demo, showing you how to use it, explaining what's under the hood, and illustrating some of the advanced techniques employed. If you are interested to know how we optimized the network transport, you can skip to the section "Techniques to Employ in the Real-Time Web Stack" below. Let's start with the link to the online demo: http://demos.lightstreamer.com/3DWorldDemo You can play with it right now. The demo can be tweaked on purpose to consume high amounts of bandwidth and CPU. For this reason, we had to put a limit on the number of connected users. If you find yourself unable to connect, please let us know, and we will try to arrange some dedicated demo time.
Table of Contents[alink=sec1]Preliminary Remarks[/alink] [alink=sec2]How to Use the Demo[/alink] [nbsp][nbsp][alink=sec3]How to Move Around[/alink] [nbsp][nbsp][alink=sec4]Your Identity[/alink] [nbsp][nbsp][alink=sec5]The Matrix[/alink] [nbsp][nbsp][alink=sec6]Tuning[/alink] [nbsp][nbsp][alink=sec7]Results[/alink] [alink=sec8]Under the Hood[/alink] [alink=sec9]Techniques to Employ in the Real-Time Web Stack[/alink] [nbsp][nbsp][alink=sec10]Dynamic Throttling[/alink] [nbsp][nbsp][alink=sec11]TCP vs. UDP[/alink] [nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][alink=sec12] Batching and Nagles' Algorithm[/alink] [nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][alink=sec13]Avoid Queuing[/alink] [nbsp][nbsp][alink=sec14]Delta Delivery[/alink] [nbsp][nbsp][alink=sec15]Stream-Sense[/alink] [nbsp][nbsp][alink=sec16]Client-to-Server in-Order Guranteed Messaging[/alink] [nbsp][nbsp][alink=sec17]Lightweight Protocol[/alink] [nbsp][nbsp][alink=sec18]Message Routing[/alink] [nbsp][nbsp][alink=sec19]Different Subscription Modes[/alink] [nbsp][nbsp][alink=sec20]Scalability[/alink] [alink=sec21]Conclusion[/alink] [alink=sec22]*Update*[/alink] [alink=sec23]Credits[/alink] [alink=sec24]Resources[/alink] [alink=sec25]References[/alink] [aname=sec1]
Preliminary RemarksWhat this demo is about:
- Lightweight 3D positional/rotational data streaming over the Web
- Bandwidth optimization and low-latency data delivery
- Dynamic throttling with adaptive data resampling for each client
- Lightweight 3D physics calculations
- Lightweight 3D rendering
- Cool 3D rendering
- In server-side mode, the physics is calculated on the server side, according to a stream of input from clients. Every change to the coordinates of each object is streamed back to the clients. This means that not only there is no prediction done on the client side, but also no interpolation! This is really an extreme scenario, where the client acts as a "dumb renderer".
How to Use the DemoPoint your browser to http://demos.lightstreamer.com/3DWorldDemo. You will see some controls and a 3D rendering of a virtual world, where some 3D objects are floating. You are the red object. Other human players are represented by blue objects, whereas white objects are played by robots (they are there to make the Default world less lonely, in case you are the only player there). http://demos.lightstreamer.com/3DWorldDemo). You will see the same world, fully synchronized. All the controls in the page have a question mark icon next to them, which provides full details on their purpose. In the Rendering box, you have two sliders for controlling the camera zoom and the camera field of view. If you close the Rendering box, you are still there, but the rendering algorithms in the client are stopped to save on CPU. [aname=sec3]HOW TO MOVE AROUND You can start moving your red object by using the keyboard. Open the Commands box to know what keys to use. Basically, you can add force impulses and torque (rotation) impulses on the three axes. If you can't use the keyboard, just press the on-screen keys with your mouse or finger. The on-screen keys are available in both the Commands and Rendering boxes. The more impulses you give, the higher the speed becomes in that direction. Eric's article. For example, let's take a single user (create a new lonely world to see the resulting bandwidth). If we deliver 33 updates/s in double-precision floating point format, when the object is translating and rotating on all three axes, the bandwidth consumption is about 5.6 KBps. In this particular kind of scenario, a fixed decimal length string with 1 or 2 decimal digits may be enough. With this encoding, the bandwidth decreases to about 3.6 KBps. But these are worst case scenarios, where all seven coordinates change (translation and rotation on all three axes). If any of these coordinates does not change, or change less frequently, Lightstreamer automatically uses a delta delivery feature to only send changed data. For example, with a single object that is only moving on the X axis, with string encoding, 2 decimals, and 33 updates/s, we get a bandwidth usage of about 1.5 KBps. More information on the optimization algorithms implemented by Lightstreamer is in the "Techniques to Employ in the Real-Time Web Stack" section below. [aname=sec8]
- JQuery for the controls
- Three.js for 3D rendering
- Lightstreamer for the real-time data transmission and for the matrix widget.
- There exists an item for each world in the game (actually, there is one item for each combination of world and representation precision, but let's keep the explanation simple). Such item works in COMMAND mode and delivers the dynamic list of players in that world, signaling players entering and leaving the world, in addition to notifying nickname changes and chat messages. The fields available for this item are: "key" (the unique identifier of each player), "command" (add, update, or delete), "nick" (the current nickname), and "msg" (the current chat message).
- For each player that enters a world, a specific item is created by the server (and subscribed by the clients) to carry all the real-time coordinates and movements for that player. This item works in MERGE mode and it is subscribed-to/unsubscribed-from by each client in that world based on the commands received as part of the first item above. The fields available for this item are: - The coordinates and the quaternions, which represent the current position of the object: "posX", "posY", "posZ", "rotX", "rotY", "rotZ", "rotW"This set of fields above is subscribed to in server-side mode, to get the actual positions in real time and render the objects accordingly. It subscribed to in client-side as well, to get the periodic authoritative resynchronizations (unless the resync period is set to 'never'). The matrix widget uses these fields to feed a Lightstreamer widget called DynaGrid. - The velocity vector and the angular momentum, which represent the current movement of the object: "Vx", "Vy", "Vz", "momx", "momy", "momz"This set of fields is subscribed to in client-side mode only, to receive the input for the client-side physics engine.
- Each client subscribes to an item in DISTINCT mode to implement presence. In other words, each player signals her presence by keeping this subscription active. By leaving the page, the automatic unsubscription determines the loss of presence, and the server can let all the other players know that the user has gone away (by leveraging the COMMAND-mode item above).
- Each client subscribes to an item in MERGE mode, to know the current downstream bandwidth (used by its own connection) in real time.
Techniques to Employ in the Real-Time Web StackSeveral techniques and features are required, as part of the real-time web stack, to help game developers implement multiplayer communication in a simple, reliable, and optimized way. Let's see some of them, delving a bit deeper. [aname=sec10]DYNAMIC THROTTLING There exists many kinds of data, which can be filtered (that is, resampled) by their very own nature. When a data feed provides a series of real-time samples for some given physical characteristics, you might want to resample such series to decrease the frequency, if you don't need all of the samples. Imagine you have a sensor that measures the current temperature 100 times per second and you want to deliver the measurements in real time to a software application. If you just want to display the value to a human user, perhaps 10 updates per second are more than enough. Or, even better, you might want to deliver as many updates as possible without saturating the link bandwidth. Resampling a real-time data flow is a complex procedure that needs to take several variables as input: the unsampled data flow, the desired maximum frequency, the desired maximum bandwidth, and the available bandwidth. Back to our 3D virtual world, when server-side physics is used, a lot of data is produced to keep all the clients in sync with the world changes. If the server-side physics works at 100 hz, in many cases, you don't need or cannot afford to send the data with the same frequency to all the clients. You want to send as many updates as you can, based on the actual bandwidth available at any time for each individual client connection. On the Developer Community pages of Valve  (the engine that powers some famous 3D games), you read:
Clients usually have only a limited amount of available bandwidth. In the worst case, players with a modem connection can't receive more than 5 to 7 KB/sec. If the server tried to send them updates with a higher data rate, packet loss would be unavoidable. Therefore, the client has to tell the server its incoming bandwidth capacity by setting the console variable rate (in bytes/second). This is the most important network variable for clients and it has to be set correctly for an optimal gameplay experience.This is done automatically and dynamically by Lightstreamer. Thanks to its adapting throttling algorithms (originated in the financial industry), Lightstreamer is able to resample the data flow for each individual user on the fly, taking into account all the dynamic variables:
- Bandwidth allocation: For each client, a maximum bandwidth can be allocated to its multiplexed stream connection. Such allocation can be changed at any time and Lightstreamer guarantees that it is always respected, whatever the original data flow bandwidth is.
- Frequency allocation: For each data flow (a subscription, in Lightstreamer terms) of each client's multiplexed stream connection, a maximum update frequency can be allocated. Again, such allocation can be changed at any time.
- Real bandwidth detection: Internet congestions are automatically detected and Lightstreamer continuously adapt to the real bandwidth available.
- X=5.1; Y=3.0; Z=2.3
- X=unchanged; Y=3.1; Z=2.5
- X=5.3; Y=unchanged; Z=2.8
Using TCP is the worst possible mistake you can make when developing a networked game! To understand why, you need to see what TCP is actually doing above IP to make everything look so simple!I'm not going to disagree with these positions, as they totally make sense, but I would like to introduce another perspective. Let's highlight the advantages of TCP versus UDP:
- It is reliable (and even if you don't always need it, it's better to have it than not, provided, of course, that the price to pay is not too high).
- If used under Web protocols (HTTP and WebSockets), it can pass through any proxy, firewall, and network intermediary.
Game data is compressed using delta compression to reduce network load. That means the server doesn't send a full world snapshot each time, but rather only changes (a delta snapshot) that happened since the last acknowledged update. With each packet sent between the client and server, acknowledge numbers are attached to keep track of their data flow. Usually full (non-delta) snapshots are only sent when a game starts or a client suffers from heavy packet loss for a couple of seconds.Delta delivery is used by default on Lightstreamer, as it has always been a mean to reduce the amount of financial data that need to be sent. If a users subscribes to 20 different fields for each stock (price, bid, ask, time, etc.), only a few of them really change at every tick. Lightstreamer automatically extracts the delta and the client-side library is able to rebuild the full state. Upon initial connection, Lightstreamer sends a full snapshot for all the subscribed items. Even in case of disconnections, the Lightstreamer client-side library automatically reconnects and restores the state via a full snapshot. [aname=sec15]STREAM-SENSE The documentation of Unity  (one of the most used 3D rendering engines) says:
Connecting servers and clients together can be a complex process. Machines can have private or public IP addresses and they can have local or external firewalls blocking access. Unity networking aims to handle as many situations as possible but there is no universal solution.Stream-Sense is an exotic name we use in Lightstreamer to refer to the ability to automatically detect the best transport protocol to use over TCP. WebSocket is the best choice, but there are many cases in the real world where WebSockets simply don't work. Apart from browser support, the real issue is with some network intermediaries (firewalls, proxies, caches, NATs, etc.) that don't speak WebSockets and block them. In all these cases, Lightstreamer automatically switches to HTTP Streaming, which is as efficient as WebSocket for sending data from the server to the client. But there are even some cases where HTTP Streaming is blocked by some corporate proxies. In such extreme situations, Lightstreamer still works by automatically switching to HTTP Smart Polling (a.k.a Long Polling). This polling style is very different from traditional polling, as the polling frequency is dynamic and data-driven. See this slide deck  for more details.
- It requires a full round trip for each message (unless HTTP pipelining is used, which is turned off by default in most browsers).
- There is no control over connection reuse. As said above, the browser has exclusive control over its connection pool and usually closes an idle connection after a timeout. So, if after some time you need to send a low-latency update to the server, you have to set up a new TCP connection first, thus increasing latency.
- It automatically batches high-frequency client messages. This means if the client tries to deliver 100 messages in a burst, instead of sending 100 HTTP requests, the browser will send just a couple of HTTP requests.
- The client messages are automatically numbered. The server can send back ACKs for each message and reorder out-of-order messages. There is mechanism in the client that controls automatic retransmission of unacknowledged messages. In other words, some sort of "TCP over HTTP" has been implemented to get reliable messaging from client to server inside a browser!
- Reverse heartbeats can be activated so the browser keeps the connection open and ready to use as soon as needed. Reverse heartbeats are tiny messages sent by the client to force the browser to avoid closing the underlying TCP connection.
d(27,1,2,'-18.85667',3,'0.91879','-0.39474',1);d(30,1,5,'-0.70090',1,'-0.71325','-59.00561');d(31,1,3,'27.42105','-0.00389','-0.44682','0.89365','-0.04139',1);d(29,1,3,'-57.12912','-0.00389','-0.44682','0.89365','-0.04139',1);d(28,1,2,'42.71763',1,'-0.40773','0.48877','-0.05785','-0.76909','-7.72828');[aname=sec18]MESSAGE ROUTING Having implemented optimized and reliable transports and protocols is not enough for developing complex real-time applications and games. On top of such mechanisms, you need an easy and flexible way to route messages, that is, to govern which messages should be delivered to which clients. Lightstreamer is based on what I call an asymmetric publish-subscribe paradigm. Clients can subscribe to items and can send messages to the server, but the actual publishers are on the server side. Based on any back-end data feed, any back-end data computation algorithms, and any message received from the clients, the publishers (Lightstreamer Data Adapters) inject real-time updates on any item into the Lightstreamer Server. Then, it's up to the server to route the updates based on the client subscriptions.
"posX", "posY", "posZ", "rotX", "rotY", "rotZ", "rotW"Let's suppose the client is interested in knowing the coordinates and the velocity vector. It will subscribe to the same item with a different schema:
"posX", "posY", "posZ", "Vx", "Vy", "Vz"If the client wants to know the nicknames and messages of the other players, in addition to be notified of players entering and leaving a world, it will subscribe to the main item (which represents that world), with a schema like this:
"nick", "msg", "key", "command"Basically, every client can subscribe to as many items it needs, each with its own schema.
ConclusionWe have presented an online demo that shows how a technology created for the financial industry can be used with great benefits in the gaming industry. Adaptive streaming (dynamic throttling, conflation, delta delivery, etc.) is a set of techniques to make sure that the amount of real-time data needed to synchronize a 3D virtual world among several clients is governed by the actual network capacity for each client. Several low-level optimizations and high-level abstractions are required to make multiplayer game development easier with reliable results. If you want to solve the problems of real-time gaming efficiently, you should use similar techniques in your real-time web stack. The source code of this demo is available on GitHub. Feel free to modify it and create any derivative work; perhaps a full-fledged game! Let us know if you find the demo code useful and if you will be working on any project based on it. We are eager for any feedback on both the demo and this article. You can contact us at firstname.lastname@example.org. [aname=sec22]
*Update*In October 2013, I had the pleasure to give a talk at HTML5 Developer Conference in San Francisco. I delved into the subject of this article and showed several live demonstrations, because seeing is believing. You can watch the full recording of the session, together with the slide deck, here: http://blog.lightstreamer.com/2013/12/html5devconf-more-than-just-websockets.html [aname=sec23]
CreditsMany thanks to Eric Li, Matt Davey, and Michael Carter for the comments. Originally publish on the Lightstreamer blog. [aname=sec24]
- Online 3D World Demo:http://demos.lightstreamer.com/3DWorldDemo
- Server-side source code of 3D World Demo:https://github.com/Weswit/Lightstreamer-example-3DWorld-adapter-java
-  Eric Li. Optimizing WebSockets Bandwidth:http://buildnewgames.com/optimizing-websockets-bandwidth/
-  Glenn Fiedler. Networked Physics:http://gafferongames.com/game-physics/networked-physics/
-  Valve Developer Community - Source Multiplayer Networkinghttps://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking
-  Chen-Chi Wu, Kuan-Ta Chen, Chih-Ming Chen, Polly Huang, and Chin-Laung Lei. On the Challenge and Design of Transport Protocols for MMORPGs:http://www.iis.sinica.edu.tw/~swc/pub/tcp_mmorpg.html
-  Glenn Fiedler. UDP vs. TCP:http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/
-  Alessandro Alinone. From Push Technology to the Real-Time Web:http://www.slideshare.net/alinone/from-push-technology-to-the-realtime-web/23
-  Unity - High Level Networking Concepts:http://docs.unity3d.com/Documentation/Components/net-HighLevelOverview.html