• Advertisement
Sign in to follow this  

Network packet unserialization (catch-22)

This topic is 2303 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm trying to work out an update to my networking system, and I've run into a catch-22 scenario.

Let's say the network code receives a stream of bytes from the server. First byte is an ID that identifies the packet. From this, the network code can either a) unserialize the packet into a struct for use by the game state, or b) pass the byte stream to the gamestate and allow it to unserialize it when needed (i.e. by encapsulating the byte array in an object that can return the reconstructed struct).

However, each of these two options has a problem.
  • If the network code unserializes the packet into a struct, it then needs to store that struct somewhere until the gamestate can use it. Since each network command has a different number of parameter types and lengths, I can't just stick all of these structs into an array. It is left with a custom struct that it doesn't know what to do with. Fail.
  • If I instead leave it as a byte array and let the gamestate reconstruct it when it's needed, I eliminate the above problem. However, there is a new problem: how do I know that the byte array is complete? Since the network code didn't try to reconstruct it, it doesn't know if it is complete. The packet may have been fragmented during transmit, or may contain additional commands appended to the end of it. The network code has already handed it off, so it won't know what to do with the remaining packet fragment when it arrives. Fail.
    I read a lot about people serializing data for transmit, and using abstraction to make their structs self-serializing. But I can't find anything on actually receiving the resulting bytes, and reconstructing the struct at the destination in a similarly friendly way. Right now I'm unserializing all commands into a single generic struct that contains all fields that any command would need. However, this is starting to become unwieldy as the number of command variations increases. Any insight on this would be great.

Share this post


Link to post
Share on other sites
Advertisement
Just include the length of the byte-array together with the ID, and you know how many bytes to wait for before the next message starts.

Share this post


Link to post
Share on other sites
Or you make the deserialisation code handle running out of bytes and resuming when given more. Then you can just hand whatever the network gives you to it. The return from the handler tells you how many bytes it used and whether the current structure was completed or not. This way the structures can even be variable in serialisation size.




It's hardly a catch-22 -- if it were, no-one else would be able to solve it either and network games would be impossible...

Share this post


Link to post
Share on other sites

Just include the length of the byte-array together with the ID, and you know how many bytes to wait for before the next message starts.

Now I feel dumb for not having considered this. I was so sure I would have to calculate the final packet length twice in order to pull this off. Thanks!



Or you make the deserialisation code handle running out of bytes and resuming when given more. Then you can just hand whatever the network gives you to it. The return from the handler tells you how many bytes it used and whether the current structure was completed or not. This way the structures can even be variable in serialisation size.

This is what I do now. The network code does the deserialization, so it is able to hold onto the buffer if the calculated packet length is greater than the available data. However, if I switch to passing the byte array to the gamestate to allow it to deserialize it directly into a final struct, at that point it is too late to make further changes.

Share this post


Link to post
Share on other sites
If the network code unserializes the packet into a struct, it then needs to store that struct somewhere until the gamestate can use it. Since each network command has a different number of parameter types and lengths, I can't just stick all of these structs into an array. It is left with a custom struct that it doesn't know what to do with. Fail.


But you can store the triple type - size - data in an array. The data itself can live elsewhere (such as allocated in a heap block).
That way, a user of a particular packet can cast the void * of your data to the proper struct (based on the type value).

Share this post


Link to post
Share on other sites

But you can store the triple type - size - data in an array. The data itself can live elsewhere (such as allocated in a heap block).
That way, a user of a particular packet can cast the void * of your data to the proper struct (based on the type value).

Good point... But what about struct padding? I always avoided casting a void* to a struct, for fear of the raw data being packed with different padding from what the platform expects.

Share this post


Link to post
Share on other sites

[quote name='hplus0603' timestamp='1319938416' post='4878373']
But you can store the triple type - size - data in an array. The data itself can live elsewhere (such as allocated in a heap block).
That way, a user of a particular packet can cast the void * of your data to the proper struct (based on the type value).

Good point... But what about struct padding? I always avoided casting a void* to a struct, for fear of the raw data being packed with different padding from what the platform expects.
[/quote]

Struct padding is under your control.
If the same compiler and CPU is used on both ends, then there will be no difference in struct padding. The platform doesn't know what the bytes mean; it only knows the size of the bytes. The interpretation of bytes as struct happens in the compiler.
If you don't use the same compiler/CPU, you can still make sure that the different versions lay out the data the same way (modulo byte order, if you go to some exotic, big-endian system :-) The compiler is fully predictable in this manner, once you know what the rules are.
Also, you may have alignment questions. If some struct member wants to be aligned on 8 bytes, say, there's no guarantee that it will be like that in your receive buffer. Thus, when taking a struct out of the receive buffer, you typically have to copy it to a memory block that is aligned on the biggest possible required alignment (which is 4, 8 or 16 bytes, typically, depending on CPU and data types used).
However, a bigger question is data structures that change in size -- strings, arrays, etc. You may want to look into "binary serialization" for solving that problem.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement