Author
Hey there;
I use C# so thats where my thinking is headed at the moment, but I figure that the basic logic of filestreams would be similar, no matter what language(please someone, correct me if I'm wrong).
Now, what I'm interested in is how people store their data. Does anyone have a really efficient way of storing data, that they are willing to share? Or is there any good tutorials on the topic - i did search the forums here, with no luck...
I haven't got any particular data type that I need to store and recover at the moment; but I have had issues with it in the recent past and this has kindled my curiosity.
Anyway, thanks in advanced!
I use a binary encoding inspired by google's protocol buffers format:
http://code.google.com/apis/protocolbuffers/docs/encoding.html
http://code.google.com/apis/protocolbuffers/docs/encoding.html
I store most of my data in XML files directly in directories below the game directory. However, I have abstracted filesystem access so that I can plug in any access scheme (currently just from any directory anywhere, like from cd, and from inside zip archives, but anything is possible, like from custom archive format or from over the network) or even chain access schemes (first look in mod dir, then in data dir, then in archive on cd, for instance).
Format wise I am pretty happy with XML, despite its verbosity. It's very easy to parse using third-party tools (or built-in ones. both Java and C# have integrated support for XML). Binary formats might be needed eventually for some types of data (especially for mesh data - only good from XML here is ease of export and checking whether my custom exporter really works).
My advise is, if you don't have a particular data type that needs storing, don't worry about this much yet, concentrate on things that will be seen instead, like gameplay or graphics or UI or physics.
Format wise I am pretty happy with XML, despite its verbosity. It's very easy to parse using third-party tools (or built-in ones. both Java and C# have integrated support for XML). Binary formats might be needed eventually for some types of data (especially for mesh data - only good from XML here is ease of export and checking whether my custom exporter really works).
My advise is, if you don't have a particular data type that needs storing, don't worry about this much yet, concentrate on things that will be seen instead, like gameplay or graphics or UI or physics.
Author
Quote:Original post by Hodgman
I use a binary encoding inspired by google's protocol buffers format:
http://code.google.com/apis/protocolbuffers/docs/encoding.html
Do you use a modified version of it? or just something that works along the same idea?
How would this work when attempting to serialize multiple objects to the same file?
Would an XNA-type wrapper be helpful?
eg:
<xml>...blah blah blah...<my_serialized_object>Serialized data</my_serialized_object><my_serialized_object>More serialized data</my_serialized_object></xml>Quote:Original post by lightbringer
However, I have abstracted filesystem access so that I can plug in any access scheme (currently just from any directory anywhere, like from cd, and from inside zip archives, but anything is possible, like from custom archive format or from over the network) or even chain access schemes (first look in mod dir, then in data dir, then in archive on cd, for instance).
Uhm... I'm not quite sure I follow. I don't know heaps about IO; aside from the basic StreamReader/StreamWriter/File classes in C#...
I have used a bit of XML before; but my issue is with its readability. I get funny about people readily being able to change stuff in the files... I know they can do it just as easily when they don't understand the files. But somehow; the whole not being able to understand it seems to make rewriting less appealing.
Quote:Original post by mickliddy
I have used a bit of XML before; but my issue is with its readability. I get funny about people readily being able to change stuff in the files... I know they can do it just as easily when they don't understand the files. But somehow; the whole not being able to understand it seems to make rewriting less appealing.
It depends I guess. I try to make it easy for the user to change things and would be happy if people were to do it - it extends the game's shelf life and means people care enough about the game to mod it :D.
Quote:
Uhm... I'm not quite sure I follow. I don't know heaps about IO; aside from the basic StreamReader/StreamWriter/File classes in C#...
One common approach in file IO is systems like PhysFS. The idea being that you abstract away the source location. Files could be in a zip/pak format for better access speed (smaller files), or in a directory for quick development editing. Files could be in Pak0 or Pak15 or PatchPak2, all you have to know is that you are looking for "models/teapot.x".
On top of that, systems like Boost::Serialization let you abstract the input and output format, so you can read/write to XML, text, or binary files. The change happens only in the code where you open the output file, and none of the code to write out your objects has to change.
Quote:Original post by mickliddyI just took inspiration from some of their binary encodings, such as the variable length integers.
Do you use a modified version of it? or just something that works along the same idea?
Quote:How would this work when attempting to serialize multiple objects to the same file?Each object in the file begins with the id/name of the class to be created. Zero-to-many of that classes variables are then written to the file (id/name of the variable + value), followed by an "end of object" marker (e.g. could be a special/reserved variable ID, meaning "no more variables").
Multiple objects can simply be written one after another to the same file. The file-loader knows that if it just opened the file, or just finished reading a root object then the next field read should be a class-ID (or end of file).
The object reader/writer class is abstracted so it can either read/write in this binary format, or in XML. The "class ID" field would be written as an XML start-tag, the "end of object" marker as an XML end-tag, and variables are either written as attributes (primitive types) or child-tags (nested objects).
e.g. Given a C++ class like this, and two objects created from it
class Widget{ std::string a, b, c; REGISTER_PROPERTY( a, 0 ) REGISTER_PROPERTY( b, 1 ) REGISTER_PROPERTY( c, 2 )};REGISTER_CLASS( Widget, 42 )Widget w1;Widget w2;w1.a = "hello";w1.c = "world";w2.b = "test";Saved as binary (Each "[value]" is actually written in binary. [-1] is the "no more variables" marker):
[42][0]["hello"][2]["world"][-1][42][1]["test"][-1]<archive> <class type="Widget" a="hello" c="world"> </class> <class type="Widget" b="test"> </class></archive>
Author
Quote:Original post by lightbringer
I try to make it easy for the user to change things and would be happy if people were to do it - it extends the game's shelf life and means people care enough about the game to mod it :D.
Well; I'm all for modding (Ive tried and failed miserably to mod a few games), but the worry I have is of people changing stuff that I don't want changed - even to mod. Eg. Files from non-game programs, save files...
Quote:Original post by Hodgman
Each object in the file begins with the id/name of the class to be created. Zero-to-many of that classes variables are then written to the file (id/name of the variable + value), followed by an "end of object" marker (e.g. could be a special/reserved variable ID, meaning "no more variables").
Multiple objects can simply be written one after another to the same file. The file-loader knows that if it just opened the file, or just finished reading a root object then the next field read should be a class-ID (or end of file).
The object reader/writer class is abstracted so it can either read/write in this binary format, or in XML. The "class ID" field would be written as an XML start-tag, the "end of object" marker as an XML end-tag, and variables are either written as attributes (primitive types) or child-tags (nested objects).
Ok; so how do I go about implementing this binary serialization in C#? Are there any disadvantages to it, compared to other storage types?
If I wanted to de/serialize an array of generic type to a file, would it simply be a case of...
public void Serialize(){ FileStream myStream = File.Open(Application.StartupPath+"\\Data\\"+myDataFile); foreach(MyType1 T in myTypeArray) { T.Serialize(myStream); } myStream.Close();}public void Deserialize(){ FileStream myStream = File.Open(Application.StartupPath+"\\Data\\"+myDataFile); myType1[] myTypeArray = myStream.Deserialize;}?
thanks!
Quote:Original post by mickliddy
Well; I'm all for modding (Ive tried and failed miserably to mod a few games), but the worry I have is of people changing stuff that I don't want changed - even to mod. Eg. Files from non-game programs, save files...
Zip em, call the file .sav or .dat or something, half the people won't be any wiser, especially if it's binary files inside. The ones who want to change stuff will roll out their hex editors eventually anyway, no way around that. I think that's the fastest solution that will get you back to working on the actual game, but you are of course free to overengineer the filesystem access as much as is needed :D
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
