Sign in to follow this  
SillyCow

Reducing mobile multiplayer traffic

Recommended Posts

Hi,

I'm programming a multiplayer VR racing game on mobile, and I have implemented a basic multiplayer server.

I've used a naiive implementation that I've previously written for a mobile strategy game.

Algorithmically everything works fine. However I am getting alot of traffic on the network because my racing game has much more frequent updates than a strategy game.

As I said, the implementation is naiive (a brute force attempt to add multiplayer in 24 hours). This means:

  • I am using TCP_NODELAY instead of UDP (can I use UDP on mobile networks? I allways assumed it was blocked)
  • I send all the other 10 players' positions to each player every 200ms. The racing is in "ghost mode". There is no interaction with the other players, you just see them on riding next to you. So 200ms update rate is not a game issue. But any slower, and the lag becomes really noticable.
  • I am using a text based delivery method (easy to debug across a C# unity client and a Java server [my old Centos server cannot run Mono] ). I am assuming this is not too bad because tcp headers are 20 bytes each anyway, and I am transporting 4 strings: 3 floats: x,y,z, and a 64 bit uuid.

Using all of this, i get around 1MB of data use per minute. Since this should run over a mobile network, that's way too high. You should assume I know very little about real-time multiplayer, with that in mind: Are there any obvious tips I should be using to reduce my traffic drastically?

My current guess is to try and reduce the text encoded messages. By replacing the 3 floats with fixed point ints (percision is not important for a ghost), and reducing the uuid to cyclical int. (If I ever have over 4 billion players, I'll be extatic to fix this bug :-) )

Edited by SillyCow

Share this post


Link to post
Share on other sites

There is something I don't understand.

I improved it a little bit by making my text packets smaller.

It is now ~50 bytes payload per player state.

So with a 200ms update I should be sending 50bytes_payload * 5messages_per_second * 60seconds = 15k bytes of payload data per minute + 6k tcp headers (300 messages).

However when I turn ff my WIFI, I see android reporting around 200k data usage per minute in the app info page. That's 10 times as much as 21k.

Is there some overhead I am missing. Could it be that I misunderstood just how bad tcp_no_delay can be?

(yes I know, the state could be smaller if it was binary encoded, but I still want to understand what I m seeing here...)

Edited by SillyCow

Share this post


Link to post
Share on other sites

can I use UDP on mobile networks?


Yes, you can (usually -- there's always some odd duck out there.)

A number of important protocols require UDP, ranging from voice-over-IP, to video streaming, to DNS look-ups, to VPN connections.

Share this post


Link to post
Share on other sites

didnt you say you send all 10 players position to each player every 200ms?  Wouldnt that explain the 10 times amount of data you were expecting?

Nope, that traffic was measured with only 2 test players connected.


Yes, you can (usually -- there's always some odd duck out there.) A number of important protocols require UDP, ranging from voice-over-IP, to video streaming, to DNS look-ups, to VPN connections.

But aren't those whitelisted? Or is any old protocol allowed? 

Share this post


Link to post
Share on other sites

is any old protocol allowed?


In general, yes. If you pay for the data, it's in the carrier's best interest for you to be able to use the data.
If you're on some "all you can eat" kind of connection, filtering might make more sense, to reduce "abuse."
Some companies also attempt to filter the common Bittorrent ports and such.

On the topic of traffic filtering: You can't really "whitelist" a protocol as an ISP/operator, because while a specific port number may be "common," people can put whatever data they want into those packets.
That being said, the reverse is also true. Some companies/enterprises used to filter traffic based on ports, so only (say) 53 UDP and 80/443 TCP would be allowed (DNS, and the web,) and games or instant messengers would then run their game traffic on port 53 or 80, to "look like" DNS or HTTP to get through those firewalls.
There also exist "deep packet inspection" firewalls that will stop a connection if it doesn't seem like it's legitimate for the port. Even SSL can be inspected to look for the SSL framing packets, especially in the beginning/setup.

Share this post


Link to post
Share on other sites

Test locally, use Wireshark to verify the data being sent. If there's some extra traffic you don't want, you'll find it quickly enough.

Regarding reducing the data sent to synchronise positions, there are a bunch of possible approaches that might work, depending on your game:

  • Batch up your updates so that you only pay the TCP/IP header cost once for several messages.
  • Racing is usually 2D, to some extent - can you simply send p, q coordinates, where p is normalised horizontal along the track (e.g. 0 is on the left barrier, 1 is on the right) and q is how far around the track you are (0 is the start line, 1 is the finish)?
  • Switch your IDs to single bytes. You only have 10 players so each player is uniquely identified by a number from 0 to 9. Those UUIDs can be sent once, at the start, if the other players truly need them.
  • If you send velocity instead of position, can you send it less often? Perhaps sending 2 velocity updates per second is enough, plus a position update every 2 seconds to prevent it drifting too far (a bit like MPEG encoding and the concept of I-Frames and P-Frames)
  • Consider using one-byte floats instead of 4-bytes or more; you probably don't need more than 255 discrete grades of speed or velocity, especially if you're sending position corrections as well.

Share this post


Link to post
Share on other sites

Can't use wireshark as my server is centos.


Wireshark runs on Linux. For example: http://serverfault.com/questions/404282/how-do-i-install-wireshark-in-red-hat-linux
Also, you can let tcpdump capture pcap files, that can then be opened in Wireshark!

That being said, Real Men (tm) use "sudo tcpdump -i any -s 65536 -XX" and call it good :-) Edited by hplus0603

Share this post


Link to post
Share on other sites

Test locally, use Wireshark to verify the data being sent. If there's some extra traffic you don't want, you'll find it quickly enough.
Regarding reducing the data sent to synchronise positions, there are a bunch of possible approaches that might work, depending on your game:

  • Batch up your updates so that you only pay the TCP/IP header cost once for several messages.
  • Racing is usually 2D, to some extent - can you simply send p, q coordinates, where p is normalised horizontal along the track (e.g. 0 is on the left barrier, 1 is on the right) and q is how far around the track you are (0 is the start line, 1 is the finish)?
  • Switch your IDs to single bytes. You only have 10 players so each player is uniquely identified by a number from 0 to 9. Those UUIDs can be sent once, at the start, if the other players truly need them.
  • If you send velocity instead of position, can you send it less often? Perhaps sending 2 velocity updates per second is enough, plus a position update every 2 seconds to prevent it drifting too far (a bit like MPEG encoding and the concept of I-Frames and P-Frames)
  • Consider using one-byte floats instead of 4-bytes or more; you probably don't need more than 255 discrete grades of speed or velocity, especially if you're sending position corrections as well.
Really good advice. I realize this is a big topic and I have read similar advice regarding reducing traffic on websocket connection.

But I am bogged down by the sheer amount of work.

Regarding the player id, I simply use the socket id as the player id but it is combined using numbers and letters which has too many bytes. But if I change it to some simpler form the id on the client side will be different from the ones on the server side as I check whether the player is connected using the socket id.

As with the floating point number, I am using 64 bits floating point which is the standard JavaScript number. It is a double-precision float. When converted to binary data its size is 8 bytes. Hopefully I can find a way to convert it to a single-precision floating point number in JavaScript.

Also JavaScript float number uses IEEE 754 standard, I hope modifying the precision won't affect the real position on browser too much.

Now each of my player sends out about 10KB/s which is far from optimized. I hope that I can reduce it to 1-2KB /s so that I can have a few hundreds happy players on the server.

Share this post


Link to post
Share on other sites
Let's back up a bit.

Why do you send the ghost players as real-time data?
You should be able to send the course of each ghost player as data ahead of time.
When you do this, you should be able to compress the data really well (gzip or something better) because you can spend whatever resources you want to compress data on the server once, when you store it.

Second, you don't need 64-bit data, at least not for ghost players, because they will just be following the trail of positions, rather than simulate in more detail.
24 bits for x and y and 16 bits for Z should get you all the precision you need, and 32 bits for a full quaternion (with one component dropped) is a common encoding.
That's 8+4 bytes per player state per update.

Also, how are you affecting the local player when they receive their own data back?
The fact that I (as local player) receive data back for the past (half a second or so back) is going to be much more "affecting the simulation" than any rounding/truncation error in the state update.
You can't do full determinism with floating point in Javascript anyway, because you don't know which FPU your client is going to run on.
If you really worry about this, then round/truncate your simulated state each time you take a simulation step, so you know the state is exactly what will be transmitted. (I e, marshal to network, then de-marshal again, as part of simulation loop.)

Anyway. Ghost tracks ahead of time. Highly compressed. Might even pre-install some good ones with the game, so players can practice offline!

Share this post


Link to post
Share on other sites

Regarding the player id, I simply use the socket id as the player id but it is combined using numbers and letters which has too many bytes. But if I change it to some simpler form the id on the client side will be different from the ones on the server side as I check whether the player is connected using the socket id.


Like I said above, if there are 10 players, each can be identified by 0 to 9. Simply tell each client:

  • "the player with Socket ID 'BlahBlahBlah' is Player 0."
  • "the player with Socket ID 'FooBar' is Player 1."
  • ...
  • "the player with Socket ID 'XYZABC' is Player 9."

You don't need to use one common ID for every system, across every computer. It's very common to have multiple 'IDs' that basically resolve to the same user - e.g. username, user number, email address, local socket port, etc. - and it's common to want to be able to take one ID and look up another, for whatever reason. You can use whichever ID is most convenient, and in this case, a simple index into an array of all your players would be effective.

Besides, I don't know what the Socket ID is in your case, but it doesn't sound like something that is guaranteed to be suitable for sharing.
 

As with the floating point number, I am using 64 bits floating point which is the standard JavaScript number. It is a double-precision float. When converted to binary data its size is 8 bytes. Hopefully I can find a way to convert it to a single-precision floating point number in JavaScript.

Also JavaScript float number uses IEEE 754 standard, I hope modifying the precision won't affect the real position on browser too much.


The conversion process between 2 types is just about scaling them up or down. It's usually easier to go via a normalised 0 to 1 representation. e.g. If I wanted to represent a number from 0 to 10000 in one byte, I might first divide by 10000 (to give a number from 0.0 to 1.0) and then multiply by 255 (to give a number between 0.0 and 255.0). I can round that to the nearest integer and then it can fit in one byte. The reverse process can expand it back out to the original 0 to 10000 range. This loses some precision, but that may not be a problem. You need to be thoughtful about where you use it, and do the mathematics to see how it would affect you.
Don't do this sort of micro-optimisation until you've tried other approaches, such as simply sending fewer messages.
 

Now each of my player sends out about 10KB/s which is far from optimized. I hope that I can reduce it to 1-2KB /s so that I can have a few hundreds happy players on the server.

 
The first thing is that you need to see exactly what this data is and work out why it is this size.
You might want to start a different topic because your needs (top-down MMO-style game with area of interest management, if I remember correctly) are very different to the original poster's needs (fast paced racing game). I suspect that you need to be sending updates less frequently, whereas the original poster may legitimately need to squash their message size down.
 

Why do you send the ghost players as real-time data?

I'm guessing the original poster is talking about a Trackmania-style game, where you race other players in real-time but can't actually interact with them in any physical sense.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this