Sign in to follow this  
mickliddy

File IO

Recommended Posts

Hey there; I use C# so thats where my thinking is headed at the moment, but I figure that the basic logic of filestreams would be similar, no matter what language(please someone, correct me if I'm wrong). Now, what I'm interested in is how people store their data. Does anyone have a really efficient way of storing data, that they are willing to share? Or is there any good tutorials on the topic - i did search the forums here, with no luck... I haven't got any particular data type that I need to store and recover at the moment; but I have had issues with it in the recent past and this has kindled my curiosity. Anyway, thanks in advanced!

Share this post


Link to post
Share on other sites
I store most of my data in XML files directly in directories below the game directory. However, I have abstracted filesystem access so that I can plug in any access scheme (currently just from any directory anywhere, like from cd, and from inside zip archives, but anything is possible, like from custom archive format or from over the network) or even chain access schemes (first look in mod dir, then in data dir, then in archive on cd, for instance).

Format wise I am pretty happy with XML, despite its verbosity. It's very easy to parse using third-party tools (or built-in ones. both Java and C# have integrated support for XML). Binary formats might be needed eventually for some types of data (especially for mesh data - only good from XML here is ease of export and checking whether my custom exporter really works).

My advise is, if you don't have a particular data type that needs storing, don't worry about this much yet, concentrate on things that will be seen instead, like gameplay or graphics or UI or physics.

Share this post


Link to post
Share on other sites
Quote:
Original post by Hodgman
I use a binary encoding inspired by google's protocol buffers format:
http://code.google.com/apis/protocolbuffers/docs/encoding.html


Do you use a modified version of it? or just something that works along the same idea?
How would this work when attempting to serialize multiple objects to the same file?
Would an XNA-type wrapper be helpful?
eg:

<xml>
...
blah blah blah
...
<my_serialized_object>
Serialized data
</my_serialized_object>
<my_serialized_object>
More serialized data
</my_serialized_object>
</xml>


Quote:
Original post by lightbringer
However, I have abstracted filesystem access so that I can plug in any access scheme (currently just from any directory anywhere, like from cd, and from inside zip archives, but anything is possible, like from custom archive format or from over the network) or even chain access schemes (first look in mod dir, then in data dir, then in archive on cd, for instance).

Uhm... I'm not quite sure I follow. I don't know heaps about IO; aside from the basic StreamReader/StreamWriter/File classes in C#...

I have used a bit of XML before; but my issue is with its readability. I get funny about people readily being able to change stuff in the files... I know they can do it just as easily when they don't understand the files. But somehow; the whole not being able to understand it seems to make rewriting less appealing.

Share this post


Link to post
Share on other sites
Quote:
Original post by mickliddy
I have used a bit of XML before; but my issue is with its readability. I get funny about people readily being able to change stuff in the files... I know they can do it just as easily when they don't understand the files. But somehow; the whole not being able to understand it seems to make rewriting less appealing.

It depends I guess. I try to make it easy for the user to change things and would be happy if people were to do it - it extends the game's shelf life and means people care enough about the game to mod it :D.

Share this post


Link to post
Share on other sites
Quote:


Uhm... I'm not quite sure I follow. I don't know heaps about IO; aside from the basic StreamReader/StreamWriter/File classes in C#...

One common approach in file IO is systems like PhysFS. The idea being that you abstract away the source location. Files could be in a zip/pak format for better access speed (smaller files), or in a directory for quick development editing. Files could be in Pak0 or Pak15 or PatchPak2, all you have to know is that you are looking for "models/teapot.x".

On top of that, systems like Boost::Serialization let you abstract the input and output format, so you can read/write to XML, text, or binary files. The change happens only in the code where you open the output file, and none of the code to write out your objects has to change.

Share this post


Link to post
Share on other sites
Quote:
Original post by mickliddy
Do you use a modified version of it? or just something that works along the same idea?
I just took inspiration from some of their binary encodings, such as the variable length integers.
Quote:
How would this work when attempting to serialize multiple objects to the same file?
Each object in the file begins with the id/name of the class to be created. Zero-to-many of that classes variables are then written to the file (id/name of the variable + value), followed by an "end of object" marker (e.g. could be a special/reserved variable ID, meaning "no more variables").

Multiple objects can simply be written one after another to the same file. The file-loader knows that if it just opened the file, or just finished reading a root object then the next field read should be a class-ID (or end of file).

The object reader/writer class is abstracted so it can either read/write in this binary format, or in XML. The "class ID" field would be written as an XML start-tag, the "end of object" marker as an XML end-tag, and variables are either written as attributes (primitive types) or child-tags (nested objects).

e.g. Given a C++ class like this, and two objects created from it
class Widget
{
std::string a, b, c;

REGISTER_PROPERTY( a, 0 )
REGISTER_PROPERTY( b, 1 )
REGISTER_PROPERTY( c, 2 )
};
REGISTER_CLASS( Widget, 42 )

Widget w1;
Widget w2;
w1.a = "hello";
w1.c = "world";
w2.b = "test";


If you wrote these objects to an archive it might look like:

Saved as binary (Each "[value]" is actually written in binary. [-1] is the "no more variables" marker):
[42][0]["hello"][2]["world"][-1][42][1]["test"][-1]
Saved as XML:
<archive>
<class type="Widget" a="hello" c="world">
</class>
<class type="Widget" b="test">
</class>
</archive>

Share this post


Link to post
Share on other sites
Quote:
Original post by lightbringer
I try to make it easy for the user to change things and would be happy if people were to do it - it extends the game's shelf life and means people care enough about the game to mod it :D.


Well; I'm all for modding (Ive tried and failed miserably to mod a few games), but the worry I have is of people changing stuff that I don't want changed - even to mod. Eg. Files from non-game programs, save files...


Quote:
Original post by Hodgman
Each object in the file begins with the id/name of the class to be created. Zero-to-many of that classes variables are then written to the file (id/name of the variable + value), followed by an "end of object" marker (e.g. could be a special/reserved variable ID, meaning "no more variables").

Multiple objects can simply be written one after another to the same file. The file-loader knows that if it just opened the file, or just finished reading a root object then the next field read should be a class-ID (or end of file).

The object reader/writer class is abstracted so it can either read/write in this binary format, or in XML. The "class ID" field would be written as an XML start-tag, the "end of object" marker as an XML end-tag, and variables are either written as attributes (primitive types) or child-tags (nested objects).


Ok; so how do I go about implementing this binary serialization in C#? Are there any disadvantages to it, compared to other storage types?
If I wanted to de/serialize an array of generic type to a file, would it simply be a case of...

public void Serialize()
{
FileStream myStream = File.Open(Application.StartupPath+"\\Data\\"+myDataFile);
foreach(MyType1 T in myTypeArray)
{
T.Serialize(myStream);
}
myStream.Close();
}

public void Deserialize()
{
FileStream myStream = File.Open(Application.StartupPath+"\\Data\\"+myDataFile);
myType1[] myTypeArray = myStream.Deserialize;
}


?

thanks!

Share this post


Link to post
Share on other sites
Quote:
Original post by mickliddy
Well; I'm all for modding (Ive tried and failed miserably to mod a few games), but the worry I have is of people changing stuff that I don't want changed - even to mod. Eg. Files from non-game programs, save files...

Zip em, call the file .sav or .dat or something, half the people won't be any wiser, especially if it's binary files inside. The ones who want to change stuff will roll out their hex editors eventually anyway, no way around that. I think that's the fastest solution that will get you back to working on the actual game, but you are of course free to overengineer the filesystem access as much as is needed :D

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this