Jump to content
  • Advertisement
Sign in to follow this  
Kitasia

[.net] Deserialization is too slow!

This topic is 3492 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello! I'm working on a level editor and long story short, when I load my levels it takes far too long. For example, I've made a very plane level ,pun intended, that is simply a bunch of triangles on a plane and it took 9 seconds to load. I used 49152 vertices which is to say 16384 triangles and the file is only 5.4 MB. I've seen games load 50MB files and take around the same amount of time. I tried making a file around 50MB and I never waited long enough to see if it would ever load. I think my wait record is around 30 minutes. Is there a better way to load files that have around the same features as serialization? These load times are terrible from what i know of load times. [Edited by - AntiGuy on January 24, 2009 9:11:48 AM]

Share this post


Link to post
Share on other sites
Advertisement
Show some code and we could give hints what you doing wrong.

Do you read a Vertex and create the object and then read the next vertex? If so either read the whole file into memory and parse it from there or read more chunks and parse that chunk.

Share this post


Link to post
Share on other sites
What kind of serialization are you using? I am assuming Xml and not Binary with speeds like those (don't get me wrong, not saying binary is THAT much faster). But even that seems too slow for such a small amount of data.

Also, use a profiler to figure out what part of the serialization takes so long. With 9 seconds on a 5.4 MB file, we know you are not I/O bound (unless something is horribly wrong). For example, what happens when you load the vertices, and how well does it scale?

Like megamoscha said, please show us them codes. :)

Share this post


Link to post
Share on other sites
Hmm.. well essentially I have a generic function for saving and loading.



public static void Save(object data, string fileNameAndPath)
{
FileStream stream = new FileStream(fileNameAndPath, FileMode.Create);
BinaryFormatter binForm = new BinaryFormatter();
binForm.Serialize(stream, data);

stream.Close();
}
public static object Load(string fileNameAndPath)
{
FileStream stream = new FileStream(fileNameAndPath, FileMode.Open);
BinaryFormatter binForm = new BinaryFormatter();

object newObject = binForm.Deserialize(stream);
stream.Close();

return newObject;
}







I really can't see what I could do wrong. All I do is serialize the entire level class at once.

Share this post


Link to post
Share on other sites
What what does the deserialization of the individual parts look like? I am guessing you have something funky going on with your vertices that is causing it to scale very poorly or create a lot of garbage on construction.

Again, profiling will likely reveal what exactly is causing the problem.

Share this post


Link to post
Share on other sites
For large volumes of data, the default serialization stuff really is slow. In several cases where I've needed faster serialization, I've used the following code:

http://www.codeproject.com/KB/cs/FastSerialization.aspx

It's really simple once you've used it (his sample explains it well). People might argue about premature optimization, but ultimately ... the standard binary formatter just isn't that fast. There is far too much per-object and per-field overhead. I still use it in many situations (especially when I'm serializing unknown data), but for types that I can control, I almost always implement ISerializable and then use his library. The speedup is just staggering.

Share this post


Link to post
Share on other sites
I don't have a profiler. The cheap ones don't work very well and the ones that do work are way out of my price range. Even if I did, I doubt it would help much.

While I said I was working on a level editor it really just hit me that that doesn't have much to do with the problem! I've serialized all types of data and anything thing that's large just takes forever to load regardless of what I'm serializing.

I believe osmanb hit the nail on the head.I'm not sure about the solution though. Most my classes are practically built on classes I can't control.

I'm using a lot of structs and classes from the XNA framework.

[Edited by - AntiGuy on January 25, 2009 4:19:20 AM]

Share this post


Link to post
Share on other sites
Just to make sure it wasn't something I was doing I did the following.



if (keyAPressed)
{
Data.Save(new byte[1000, 1000,30], "example");
}
if (keyBPressed)
{
Data.Load("example");
}




The output was 28.6 MB and it took about 16 seconds to save and 16 seconds to load.

To compare times I loaded a 49MB level in UnrealEd 3.0. That took around 4 seconds. It can't really be 4x slower on something that doesn't even have complex data.

Share this post


Link to post
Share on other sites
Almost 29MB for no data? I threw together the following:


public class Data
{
private List<Vector3> test;
private Random _random;

public List<Vector3> Test
{
get { return test; }
set { test = value; }
}

public Data()
{
test = new List<Vector3>();
_random = new Random();

for (int i = 0; i < 50000; i++)
test.Add(new Vector3(_random.Next(0, byte.MaxValue), _random.Next(0, byte.MaxValue), _random.Next(0, byte.MaxValue)));

}
}

public class Game1 : Microsoft.Xna.Framework.Game
{
GraphicsDeviceManager graphics;
SpriteBatch spriteBatch;

private KeyboardState _lastState;

private Data _data;

public Game1()
{
graphics = new GraphicsDeviceManager(this);
Content.RootDirectory = "Content";
}

protected override void Initialize()
{
_data = new Data();


base.Initialize();
}

protected override void Update(GameTime gameTime)
{

KeyboardState state = Keyboard.GetState();

long start, end;

if (state.IsKeyDown(Keys.A) && _lastState.IsKeyUp(Keys.A))
{
start = DateTime.Now.Ticks;

XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = true;

XmlWriter writer;

writer = XmlWriter.Create("data.xml", settings);

IntermediateSerializer.Serialize(writer, _data, null);

writer.Close();

end = DateTime.Now.Ticks;

TimeSpan elapsedSpan = new TimeSpan(end-start);
Console.WriteLine(elapsedSpan.Milliseconds.ToString());
}
else if (state.IsKeyDown(Keys.B) && _lastState.IsKeyUp(Keys.B))
{
start = DateTime.Now.Ticks;

_data = LoadData();

end = DateTime.Now.Ticks;

TimeSpan elapsedSpan = new TimeSpan(end - start);
Console.WriteLine(elapsedSpan.Milliseconds.ToString());
}

_lastState = state;

base.Update(gameTime);
}

private Data LoadData()
{
try
{
XmlReader reader = XmlReader.Create(new FileStream("data.xml", FileMode.Open));

Data data = IntermediateSerializer.Deserialize<Data>(reader, null);

reader.Close();

return data;
}
catch (Exception ex)
{
return null;
}
}
}






The write result was 281 milliseconds and a 523K file. Read result was 484 milliseconds.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!