Jump to content
Site Stability Read more... ×
  • Advertisement

EarthBanana

Member
  • Content Count

    265
  • Joined

  • Last visited

Community Reputation

1798 Excellent

About EarthBanana

  • Rank
    Member

Personal Information

Social

  • Github
    dprandle

Recent Profile Visitors

14032 profile views
  1. Introduction Serialization is the process of taking structures and objects along with their states and converting them to data that is reproducable with any computer environment. There are many ways to do this - it can be a struggle to figure out something that consistantly works. Here I will talk about a pattern or method I like to call pupping, and it is very similar to the method boost uses for its serialization library. Packing and Unpacking Pup stands for pack-unpack, and so pupping is packing/unpacking. The ideas is this; rather than create serialize/de-serialize functions for each object type and/or each type of medium (file, network, GUI widgets), create pupper objects for each type of medium bundled with a set of read/write functions for each fundamental data type. A pupper object contains the data and functions necessary to handle reading/writing to/from the specific medium. For example, a binary file pupper might contain a file stream that each pup function would use to read/write from/to. Explanation This pattern is fairly similar boost serialization, though I was using it before hearing of boost. It is useful in any case to understand it and possibly use a custom implementation so that no boost dependency is needed. The "pupper" is somewhat equivalent to the boost archive, and pup functions are equivalent to boost serialize functions. The code presented here is more simple than boost, and does not overload operators as boost does. It is as non-invasive as possible, and not template heavy. The idea is that any object, no matter how complex, can be serialized to a stream of bytes by recursively breaking down the object until reaching fundamental data types. Any fundamental data type can be directly represented by bytes. The process of saving/loading/transmitting the raw data is separable from serializing objects. It is only necessary, then, to write the code to serialize an object once and anything can be done with the raw data. The pupper pattern differs from most serialization methods in a few ways: 1) Read and write operations are not separated except at the lowest level (in the pupper object) 2) Objects that need to be serialized do not need to inherit from any special classes 3) Can be implemented with very small overhead, using no external libraries, while remaining extendable and flexible 4) Writing class methods, virtual or otherwise, is largely not necessary If polymorphic serialization is required, a virtual method is needed in base classes. CRTP can be used to aid in this process. This case is covered later. Instead of creating a class method in each object to provide for serialization, a global function is created for each object. All global functions should have the same name and parameters, except the parameter for the object that should be serialized. Making any object "serializable" is then just a matter of writing a global function. These functions can be named whatever as long as they all have the same name, but I find "pup" fitting. Some examples of pup prototypes for stl containers are shown below. template pup(pupper * p, std::map & map, const var_info & info); template pup(pupper * p, std::vector & vec, const var_info & info); template pup(pupper * p, std::set & set, const var_info & info); The pupper pointer and var_info reference parameters will be explained later. The important thing is that the serialization work is done in a global function, not a member function. The pup pattern is easiest shown by example. In this article pupper objects for binary and text file saving/loading are coded, and a few example objects are saved/loaded using them. An example is given using the pupper pattern along with CRTP to serialize polymorphic objects. Also, a std::vector of polymorphic base objects is saved/loaded illustrating the flexibility this pattern allows when using other library defined types (std::vector). So without further ado, take a look at the pupper header file. #define PUP_OUT 1 #define PUP_IN 2 #include #include #include struct var_info { var_info(const std::string & name_): name(name_) {} virtual ~var_info() {} std::string name; }; struct pupper { pupper(int32_t io_): io(io_) {} virtual ~pupper() {} virtual void pup(char & val_, const var_info & info_) = 0; virtual void pup(wchar_t & val_, const var_info & info_) = 0; virtual void pup(int8_t & val_, const var_info & info_) = 0; virtual void pup(int16_t & val_, const var_info & info_) = 0; virtual void pup(int32_t & val_, const var_info & info_) = 0; virtual void pup(int64_t & val_, const var_info & info_) = 0; virtual void pup(uint8_t & val_, const var_info & info_) = 0; virtual void pup(uint16_t & val_, const var_info & info_) = 0; virtual void pup(uint32_t & val_, const var_info & info_) = 0; virtual void pup(uint64_t & val_, const var_info & info_) = 0; virtual void pup(float & val_, const var_info & info_) = 0; virtual void pup(double & val_, const var_info & info_) = 0; virtual void pup(long double & val_, const var_info & info_) = 0; virtual void pup(bool & val_, const var_info & info_) = 0; int32_t io; }; void pup(pupper * p, char & val_, const var_info & info_); void pup(pupper * p, wchar_t & val_, const var_info & info_); void pup(pupper * p, int8_t & val_, const var_info & info_); void pup(pupper * p, int16_t & val_, const var_info & info_); void pup(pupper * p, int32_t & val_, const var_info & info_); void pup(pupper * p, int64_t & val_, const var_info & info_); void pup(pupper * p, uint8_t & val_, const var_info & info_); void pup(pupper * p, uint16_t & val_, const var_info & info_); void pup(pupper * p, uint32_t & val_, const var_info & info_); void pup(pupper * p, uint64_t & val_, const var_info & info_); void pup(pupper * p, float & val_, const var_info & info_); void pup(pupper * p, double & val_, const var_info & info_); void pup(pupper * p, long double & val_, const var_info & info_); void pup(pupper * p, bool & val_, const var_info & info_); A var_info struct is declared first which simply has a name field for now - this is where information about the pupped variable belongs. It is filled out during the pupping process, and so a constructor requiring field information is made so that it isn't later forgotten. The pupper base class defines the set of methods that any type of pupper must implement - a method to handle reading/writing each fundamental data type from/to the medium. A set of global functions named "pup" are declared and defined, establishing the fundamental usage of the pupping pattern. The idea is to be able to call pup(pupper, object, description) almost anywhere in code in order to serialize/de-serialize any object (that should be serializable). Creating a new pupper object type includes implementing a pup method for each fundamental data type. These methods are then used by the pup global functions, which in turn are used by pup functions for more complicated types. No matter how many new pupper types are created, the pup functions to serialize each object need only be written once. This is exactly what makes this pattern useful. To make all objects serializable to file in binary, create a binary file pupper. To make all objects serializable to file in text, create a text file pupper. To make all objects serializable to a Qt dialog, create a Qt dialog pupper. Some types of pupper objects may require additional information about the variables. For example, there are multiple ways a double can be represented in a GUI - a vertical slider, horizontal slider, spin box, etc. The var_info struct allows new information about variables to be added. Any pupper object that does not need that information can just ignore it. With the Qt example, a flag could be added to the var_info struct and used by the Qt pupper object. The objects that need to be shown in a GUI would then need to set the flag, and all pupper objects that don't have use for the flag ignore it. By making the destructor of var_info virtual, the var_info struct can be extended. This is useful, again, if creating a library that others will be using. It allows the user to create their own pupper object types and add any necessary data to var_info without needing to edit the library source code. There are a few reasons for using pup(pupper, object, description) instead of pupper->pup(object, description) or object->pup(pupper, description). The reasons for not using pupper->pup(object, description) are: 1) The base pupper class would have to be extended for every new type of object. If creating a library with extendable classes, the user of the library would have to edit the base pupper class for every class they extended in which the library is still responsible for serializing 2) The pack/unpack code would be separated from the object making it prone to bugs when changes are made to the object And the reasons for not using object->pup(pupper, description) are: 1) You cannot easily extend third party library objects (such as std::vector) to include a pup function - they would require a special function or wrapper class 2) Since many objects would not include a "pup" function, there would be inconsistencies with the pup usage. This is purely an aesthetics/convenience argument, and is of course an opinion. But I would argue that writing: pup(pupper,obj1,desc1); pup(pupper,obj2,desc2); pup(pupper,obj3,desc3); pup(pupper,obj4,desc4); //etc... is both easier to understand and remember than: obj1->pup(pupper,desc1); pup(pupper,obj2,desc2); obj3->pup(pupper,desc3); pup(pupper,obj4,desc4); //etc... If the same pup function format is used for everything, writing pup functions becomes trivial because they are just combinations of other pup functions of the same format. Creating concrete pupper objects can be easy - binary and text file pupper objects are included as an example. The definition code for them is boring so it won't be shown here - but the declarations are below. //binary_file_pupper header #include "pupper.h" struct binary_file_pupper : public pupper { binary_file_pupper(std::fstream & fstrm, int mode); std::fstream & fs; void pup(char & val_, const var_info & info_); void pup(wchar_t & val_, const var_info & info_); void pup(int8_t & val_, const var_info & info_); void pup(int16_t & val_, const var_info & info_); void pup(int32_t & val_, const var_info & info_); void pup(int64_t & val_, const var_info & info_); void pup(uint8_t & val_, const var_info & info_); void pup(uint16_t & val_, const var_info & info_); void pup(uint32_t & val_, const var_info & info_); void pup(uint64_t & val_, const var_info & info_); void pup(float & val_, const var_info & info_); void pup(double & val_, const var_info & info_); void pup(long double & val_, const var_info & info_); void pup(bool & val_, const var_info & info_); }; template void pup_bytes(binary_file_pupper * p, T & val_) { if (p->io == PUP_IN) p->fs.read((char*)&val_, sizeof(T)); else p->fs.write((char*)&val_, sizeof(T)); } //text_file_pupper header #include "pupper.h" struct text_file_pupper : public pupper { text_file_pupper(std::fstream & fstrm, int mode); std::fstream & fs; void pup(char & val_, const var_info & info_); void pup(wchar_t & val_, const var_info & info_); void pup(int8_t & val_, const var_info & info_); void pup(int16_t & val_, const var_info & info_); void pup(int32_t & val_, const var_info & info_); void pup(int64_t & val_, const var_info & info_); void pup(uint8_t & val_, const var_info & info_); void pup(uint16_t & val_, const var_info & info_); void pup(uint32_t & val_, const var_info & info_); void pup(uint64_t & val_, const var_info & info_); void pup(float & val_, const var_info & info_); void pup(double & val_, const var_info & info_); void pup(long double & val_, const var_info & info_); void pup(bool & val_, const var_info & info_); }; template void pup_text(text_file_pupper * p, T val, const var_info & info, std::string & line) { std::string begtag, endtag; begtag = ""; endtag = ""; if (p->io == PUP_OUT) { p->fs
  2. EarthBanana

    Pupping - a method for serializing data

      Yeah you really can make it arbitrarily complicated depending on what's needed - for example I used this method to write network code for integrated systems that ended up being much more complicated than what is shown here.   I am trying to edit the code to clear up the spacing issue but I'm having troubles - it shows correctly in edit mode so I think some invisible formatting stuff has carried over from copy pasting - I might end just going through line by line and have the auto indent in the gamedev editor do the indenting.   Thanks for the feedback! 
  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!