• Advertisement
Sign in to follow this  

[.net] Deserialization is too slow!

This topic is 3309 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello! I'm working on a level editor and long story short, when I load my levels it takes far too long. For example, I've made a very plane level ,pun intended, that is simply a bunch of triangles on a plane and it took 9 seconds to load. I used 49152 vertices which is to say 16384 triangles and the file is only 5.4 MB. I've seen games load 50MB files and take around the same amount of time. I tried making a file around 50MB and I never waited long enough to see if it would ever load. I think my wait record is around 30 minutes. Is there a better way to load files that have around the same features as serialization? These load times are terrible from what i know of load times. [Edited by - AntiGuy on January 24, 2009 9:11:48 AM]

Share this post


Link to post
Share on other sites
Advertisement
Show some code and we could give hints what you doing wrong.

Do you read a Vertex and create the object and then read the next vertex? If so either read the whole file into memory and parse it from there or read more chunks and parse that chunk.

Share this post


Link to post
Share on other sites
What kind of serialization are you using? I am assuming Xml and not Binary with speeds like those (don't get me wrong, not saying binary is THAT much faster). But even that seems too slow for such a small amount of data.

Also, use a profiler to figure out what part of the serialization takes so long. With 9 seconds on a 5.4 MB file, we know you are not I/O bound (unless something is horribly wrong). For example, what happens when you load the vertices, and how well does it scale?

Like megamoscha said, please show us them codes. :)

Share this post


Link to post
Share on other sites
Hmm.. well essentially I have a generic function for saving and loading.



public static void Save(object data, string fileNameAndPath)
{
FileStream stream = new FileStream(fileNameAndPath, FileMode.Create);
BinaryFormatter binForm = new BinaryFormatter();
binForm.Serialize(stream, data);

stream.Close();
}
public static object Load(string fileNameAndPath)
{
FileStream stream = new FileStream(fileNameAndPath, FileMode.Open);
BinaryFormatter binForm = new BinaryFormatter();

object newObject = binForm.Deserialize(stream);
stream.Close();

return newObject;
}







I really can't see what I could do wrong. All I do is serialize the entire level class at once.

Share this post


Link to post
Share on other sites
What what does the deserialization of the individual parts look like? I am guessing you have something funky going on with your vertices that is causing it to scale very poorly or create a lot of garbage on construction.

Again, profiling will likely reveal what exactly is causing the problem.

Share this post


Link to post
Share on other sites
For large volumes of data, the default serialization stuff really is slow. In several cases where I've needed faster serialization, I've used the following code:

http://www.codeproject.com/KB/cs/FastSerialization.aspx

It's really simple once you've used it (his sample explains it well). People might argue about premature optimization, but ultimately ... the standard binary formatter just isn't that fast. There is far too much per-object and per-field overhead. I still use it in many situations (especially when I'm serializing unknown data), but for types that I can control, I almost always implement ISerializable and then use his library. The speedup is just staggering.

Share this post


Link to post
Share on other sites
I don't have a profiler. The cheap ones don't work very well and the ones that do work are way out of my price range. Even if I did, I doubt it would help much.

While I said I was working on a level editor it really just hit me that that doesn't have much to do with the problem! I've serialized all types of data and anything thing that's large just takes forever to load regardless of what I'm serializing.

I believe osmanb hit the nail on the head.I'm not sure about the solution though. Most my classes are practically built on classes I can't control.

I'm using a lot of structs and classes from the XNA framework.

[Edited by - AntiGuy on January 25, 2009 4:19:20 AM]

Share this post


Link to post
Share on other sites
Just to make sure it wasn't something I was doing I did the following.



if (keyAPressed)
{
Data.Save(new byte[1000, 1000,30], "example");
}
if (keyBPressed)
{
Data.Load("example");
}




The output was 28.6 MB and it took about 16 seconds to save and 16 seconds to load.

To compare times I loaded a 49MB level in UnrealEd 3.0. That took around 4 seconds. It can't really be 4x slower on something that doesn't even have complex data.

Share this post


Link to post
Share on other sites
Almost 29MB for no data? I threw together the following:


public class Data
{
private List<Vector3> test;
private Random _random;

public List<Vector3> Test
{
get { return test; }
set { test = value; }
}

public Data()
{
test = new List<Vector3>();
_random = new Random();

for (int i = 0; i < 50000; i++)
test.Add(new Vector3(_random.Next(0, byte.MaxValue), _random.Next(0, byte.MaxValue), _random.Next(0, byte.MaxValue)));

}
}

public class Game1 : Microsoft.Xna.Framework.Game
{
GraphicsDeviceManager graphics;
SpriteBatch spriteBatch;

private KeyboardState _lastState;

private Data _data;

public Game1()
{
graphics = new GraphicsDeviceManager(this);
Content.RootDirectory = "Content";
}

protected override void Initialize()
{
_data = new Data();


base.Initialize();
}

protected override void Update(GameTime gameTime)
{

KeyboardState state = Keyboard.GetState();

long start, end;

if (state.IsKeyDown(Keys.A) && _lastState.IsKeyUp(Keys.A))
{
start = DateTime.Now.Ticks;

XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = true;

XmlWriter writer;

writer = XmlWriter.Create("data.xml", settings);

IntermediateSerializer.Serialize(writer, _data, null);

writer.Close();

end = DateTime.Now.Ticks;

TimeSpan elapsedSpan = new TimeSpan(end-start);
Console.WriteLine(elapsedSpan.Milliseconds.ToString());
}
else if (state.IsKeyDown(Keys.B) && _lastState.IsKeyUp(Keys.B))
{
start = DateTime.Now.Ticks;

_data = LoadData();

end = DateTime.Now.Ticks;

TimeSpan elapsedSpan = new TimeSpan(end - start);
Console.WriteLine(elapsedSpan.Milliseconds.ToString());
}

_lastState = state;

base.Update(gameTime);
}

private Data LoadData()
{
try
{
XmlReader reader = XmlReader.Create(new FileStream("data.xml", FileMode.Open));

Data data = IntermediateSerializer.Deserialize<Data>(reader, null);

reader.Close();

return data;
}
catch (Exception ex)
{
return null;
}
}
}






The write result was 281 milliseconds and a 523K file. Read result was 484 milliseconds.

Share this post


Link to post
Share on other sites
Quote:
Original post by Machaira
Almost 29MB for no data? I threw together the following:

*** Source Snippet Removed ***


Your code uses the IntermediateSerializer class which is dependent on the Microsoft.Xna.Framework.Content.Pipeline assembly. If I remember correctly, the EULA forbids distributing that assembly with your game. Also, the timing code should use the TotalMilliseconds property not the Milliseconds property.

I tried your example with the BinaryFormatter as the original poster used and it took 453ms to save and 578ms to load on an archaic single core AMD machine built in early 2004. The resulting file was 1.3MB.


using System;
using System.Collections.Generic;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;

[Serializable]
public class Data
{
private readonly Random _random;
private List<Vector3> test;

public Data()
{
test = new List<Vector3>();
_random = new Random();

for (int i = 0; i < 50000; i++)
test.Add(new Vector3(_random.Next(0, byte.MaxValue), _random.Next(0, byte.MaxValue), _random.Next(0, byte.MaxValue)));
}

public List<Vector3> Test
{
get { return test; }
set { test = value; }
}
}

public class Game1 : Game
{
private Data _data;
private KeyboardState _lastState;
private GraphicsDeviceManager graphics;
private SpriteBatch spriteBatch;

public Game1()
{
graphics = new GraphicsDeviceManager(this);
Content.RootDirectory = "Content";
}

protected override void Initialize()
{
_data = new Data();


base.Initialize();
}

protected override void Update(GameTime gameTime)
{
KeyboardState state = Keyboard.GetState();

long start, end;

if (state.IsKeyDown(Keys.A) && _lastState.IsKeyUp(Keys.A))
{
start = DateTime.Now.Ticks;

Save(_data, @"c:\xxx.xxx");

end = DateTime.Now.Ticks;

var elapsedSpan = new TimeSpan(end - start);
Console.WriteLine(elapsedSpan.TotalMilliseconds.ToString());
}
else if (state.IsKeyDown(Keys.B) && _lastState.IsKeyUp(Keys.B))
{
start = DateTime.Now.Ticks;

_data = Load(@"c:\xxx.xxx") as Data;

end = DateTime.Now.Ticks;

var elapsedSpan = new TimeSpan(end - start);
Console.WriteLine(elapsedSpan.TotalMilliseconds.ToString());
}

_lastState = state;

base.Update(gameTime);
}

public static void Save(object data, string fileNameAndPath)
{
var stream = new FileStream(fileNameAndPath, FileMode.Create);
var binForm = new BinaryFormatter();
binForm.Serialize(stream, data);

stream.Close();
}

public static object Load(string fileNameAndPath)
{
var stream = new FileStream(fileNameAndPath, FileMode.Open);
var binForm = new BinaryFormatter();

object newObject = binForm.Deserialize(stream);
stream.Close();

return newObject;
}
}




Yet when I replace this line:
Quote:

Save(_data, @"c:\xxx.xxx");

with this:
Quote:

Save(new byte[1000, 1000, 30], @"c:\example.dat");

it takes 22593 ms on my machine and results in the 29MB file. It looks like the BinaryFormatter doesn't work very well with multidimensional arrays.

Changing the array of bytes to a single dimensional array results in a 650ms write and a 60 ms read.

Share this post


Link to post
Share on other sites
This doesn't really solve the problem of slow deserialization but I think I've found somewhat of a workaround. I took the larget amount of data (the vertex data), marked it as nonserializable and used the binary writer to write it all at the end of the file after serializing the level. This wasn't a problem because I doubt I'll ever change the vertex data again.

The 9 second load was cut down to a tad over a second and the file has shrunk down to 2.86MB while the level remains the exact same : )!

I still wonder if someone's found a more universal solution...

Share this post


Link to post
Share on other sites
Quote:
Original post by cyansoft
Your code uses the IntermediateSerializer class which is dependent on the Microsoft.Xna.Framework.Content.Pipeline assembly. If I remember correctly, the EULA forbids distributing that assembly with your game.

Correct. But he stated he was working on a level editor, not a game. The actual game would use the files produced by the editor with the content pipeline.

Quote:
Original post by cyansoft
Also, the timing code should use the TotalMilliseconds property not the Milliseconds property.

You're correct. Doing this takes about 4421.875 milliseconds to save and 296.875 to load. I get caught by that a lot it seems. [sad]


Changing the list to 500000 elements takes 5359.375 to save and 3546.875 to load. The file is 5229K.

This is on a P4 2.8 GHz with 1.25 GB of RAM. This is as in debug mode. I'm sure the hard drive isn't the fastest in the work either since this is a work machine.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement