# Binary serialization and queuing

This topic is 2953 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hello (me again),

I was using json AND sending way too much datas.

I know that json is not evil but isn't binary serialization still a bit better for realtime networking ?

So after reducing the size of the datas i'm sending i decided to switch to binary serialization and one thing i don't understand is how you are queuing datas, here is a sample in python :

 PLAYER_UPDATE = 1 p_update_struct = struct.Struct("ii") tosend = "" def send(msg_id, mystruct): global tosend tosend += "@"+str(msg_id)+mystruct data = p_update_struct.pack(10,20) send(PLAYER_UPDATE, data) data = p_update_struct.pack(30,20) send(PLAYER_UPDATE, data)

Basically the idea is to use a separator like this [SEPARATOR][MESSAGE_ID][BINARYDATA] so :
1 ) It's ugly
2 ) It doesn't work since @ can also be the first character of BINARYDATA and mess all the decoding.

What am i missing ?

##### Share on other sites

Basically the idea is to use a separator like this [SEPARATOR][MESSAGE_ID][BINARYDATA] so :
1 ) It's ugly
2 ) It doesn't work since @ can also be the first character of BINARYDATA and mess all the decoding.

What am i missing ?

Variable-size data should generally have 1) a maximum size and 2) a prefix value of length.
Thus, to send a string that can be between 0 and 255 characters in length, send a byte with the length, and then that many bytes of string data.
To send a binary blob that can be between 0 and 65535 characters in length, send a halfword (2 bytes, "unsigned short" or "uint16") with the length, and then that many bytes of binary data.
As long as the other end knows the order of the fields ("first comes a string, then a binary data, then ...") then it can decode the incoming data just fine.

##### Share on other sites
Don't use strings to send data. Use a completely binary solution.

I'd read this article first which should give you a general idea of how to structure a binary packet. It covers using a bit writer/reader, but the ideas also work for a byte writer/reader. One might already be built into python?

Also I'm curious is send actually sending a packet? In real-time networking you want to send at most 1 packet to a user per server update. (Also the easiest thing when you're just beginning is to use a fixed time update of like 100 ms or something) So your update on the server ends up looking like:

update() { ProcessIncomingPackets(); UpdateGameWorld(); foreach client { Write everything the player needs to know about to a packet and send it. } }

##### Share on other sites

Don't use strings to send data. Use a completely binary solution.

Strings are binary data in Python.

Also I'm curious is send actually sending a packet? In real-time networking you want to send at most 1 packet to a user per server update.[/quote]
No it doesn't directly send it. I'm actually, in my example, appending the variable "tosend" which is sent at every network tick.

Variable-size data should generally have 1) a maximum size and 2) a prefix value of length.
Thus, to send a string that can be between 0 and 255 characters in length, send a byte with the length, and then that many bytes of string data.
To send a binary blob that can be between 0 and 65535 characters in length, send a halfword (2 bytes, "unsigned short" or "uint16") with the length, and then that many bytes of binary data.
As long as the other end knows the order of the fields ("first comes a string, then a binary data, then ...") then it can decode the incoming data just fine.
Understood, thanks !

So since i'm looking for compactness I guess i have to put all the not-so-often-updated-player-states, like where my character is facing (left/right in a 2D world), in a separate binary object !?

##### Share on other sites

[quote name='Sirisian' timestamp='1316584497' post='4864115']
Don't use strings to send data. Use a completely binary solution.

Strings are binary data in Python.
[/quote]

Basically the idea is to use a separator like this [SEPARATOR][MESSAGE_ID][BINARYDATA] so :
1 ) It's ugly
2 ) It doesn't work since @ can also be the first character of BINARYDATA and mess all the decoding.

My point was that you should use event codes to separate things.
[Event ID][Data for that Event][Event ID][Data for that Event]

So if you send player updates to a client it'll end up looking like:
[Event ID: EntityUpdates][ushort: Number of entities]([ushort: entity id][byte: entity type][position... and other stuff])*

If you want a really optimal way you'll want to read this. With that design your server only sends what the clients need to know and the format turns into binary packets with only:

[Event ID: EntityUpdates] [ushort: Number of full state entities aka new entities for the client] ([ushort: entity id] [byte: entity type] [The Full State data. Everything to construct the entity with that type from scratch. Think factory pattern. EntityFactory.Create(typeID, packet)])* [ushort: Number of delta state entities aka known entities for the client] ([ushort: entity id] [delta information. The client already knows the type and the full state. Tell it what changed])* [ushort: Number of entities to forget] ([ushort: entity id])*

Ideally your parser becomes something as trivial as a look-up table with binded function for each event id or a simple switch statement on the client and server side.

##### Share on other sites

So since i'm looking for compactness I guess i have to put all the not-so-often-updated-player-states, like where my character is facing (left/right in a 2D world), in a separate binary object !?

I'd expect facing to be changing often in most games, but whatever.

If you only send the full state of each object when you send something, then yes, you should factor your game to put things that change together, together in an object.

Most systems will support updating only particular properties of a particular object. Thus, you can for example number each property from 0 to 255, and send a property change message; something like:

<propertychange><objectid><propertyid><value>

You could also crunch all updates into a single message:
<propertychanges><numobjects><objectid><numprops><propertyid><value><propertyid><value>...<objectid><propertyid><value>...

If you have fewer properties, you can stuff them into a bit mask instead of an ID per each -- this is somewhat akin to what the Quake 3 network model does with its "delta states" where it sends all properties that have changed since the last acknowledged state from the other end, and uses a bitmask to figure out which states are included.

• ### Game Developer Survey

We are looking for qualified game developers to participate in a 10-minute online survey. Qualified participants will be offered a \$15 incentive for your time and insights. Click here to start!

• 15
• 12
• 9
• 11
• 15