[.net] Deserialization is too slow!
Author
Hello! I'm working on a level editor and long story short, when I load my levels it takes far too long.
For example, I've made a very plane level ,pun intended, that is simply a bunch of triangles on a plane and it took 9 seconds to load. I used 49152 vertices which is to say 16384 triangles and the file is only 5.4 MB. I've seen games load 50MB files and take around the same amount of time. I tried making a file around 50MB and I never waited long enough to see if it would ever load. I think my wait record is around 30 minutes.
Is there a better way to load files that have around the same features as serialization? These load times are terrible from what i know of load times.
[Edited by - AntiGuy on January 24, 2009 9:11:48 AM]
Show some code and we could give hints what you doing wrong.
Do you read a Vertex and create the object and then read the next vertex? If so either read the whole file into memory and parse it from there or read more chunks and parse that chunk.
Do you read a Vertex and create the object and then read the next vertex? If so either read the whole file into memory and parse it from there or read more chunks and parse that chunk.
What kind of serialization are you using? I am assuming Xml and not Binary with speeds like those (don't get me wrong, not saying binary is THAT much faster). But even that seems too slow for such a small amount of data.
Also, use a profiler to figure out what part of the serialization takes so long. With 9 seconds on a 5.4 MB file, we know you are not I/O bound (unless something is horribly wrong). For example, what happens when you load the vertices, and how well does it scale?
Like megamoscha said, please show us them codes. :)
Also, use a profiler to figure out what part of the serialization takes so long. With 9 seconds on a 5.4 MB file, we know you are not I/O bound (unless something is horribly wrong). For example, what happens when you load the vertices, and how well does it scale?
Like megamoscha said, please show us them codes. :)
Author
Hmm.. well essentially I have a generic function for saving and loading.
I really can't see what I could do wrong. All I do is serialize the entire level class at once.
public static void Save(object data, string fileNameAndPath) { FileStream stream = new FileStream(fileNameAndPath, FileMode.Create); BinaryFormatter binForm = new BinaryFormatter(); binForm.Serialize(stream, data); stream.Close(); } public static object Load(string fileNameAndPath) { FileStream stream = new FileStream(fileNameAndPath, FileMode.Open); BinaryFormatter binForm = new BinaryFormatter(); object newObject = binForm.Deserialize(stream); stream.Close(); return newObject; }I really can't see what I could do wrong. All I do is serialize the entire level class at once.
What what does the deserialization of the individual parts look like? I am guessing you have something funky going on with your vertices that is causing it to scale very poorly or create a lot of garbage on construction.
Again, profiling will likely reveal what exactly is causing the problem.
Again, profiling will likely reveal what exactly is causing the problem.
For large volumes of data, the default serialization stuff really is slow. In several cases where I've needed faster serialization, I've used the following code:
http://www.codeproject.com/KB/cs/FastSerialization.aspx
It's really simple once you've used it (his sample explains it well). People might argue about premature optimization, but ultimately ... the standard binary formatter just isn't that fast. There is far too much per-object and per-field overhead. I still use it in many situations (especially when I'm serializing unknown data), but for types that I can control, I almost always implement ISerializable and then use his library. The speedup is just staggering.
http://www.codeproject.com/KB/cs/FastSerialization.aspx
It's really simple once you've used it (his sample explains it well). People might argue about premature optimization, but ultimately ... the standard binary formatter just isn't that fast. There is far too much per-object and per-field overhead. I still use it in many situations (especially when I'm serializing unknown data), but for types that I can control, I almost always implement ISerializable and then use his library. The speedup is just staggering.
Author
I don't have a profiler. The cheap ones don't work very well and the ones that do work are way out of my price range. Even if I did, I doubt it would help much.
While I said I was working on a level editor it really just hit me that that doesn't have much to do with the problem! I've serialized all types of data and anything thing that's large just takes forever to load regardless of what I'm serializing.
I believe osmanb hit the nail on the head.I'm not sure about the solution though. Most my classes are practically built on classes I can't control.
I'm using a lot of structs and classes from the XNA framework.
[Edited by - AntiGuy on January 25, 2009 4:19:20 AM]
While I said I was working on a level editor it really just hit me that that doesn't have much to do with the problem! I've serialized all types of data and anything thing that's large just takes forever to load regardless of what I'm serializing.
I believe osmanb hit the nail on the head.I'm not sure about the solution though. Most my classes are practically built on classes I can't control.
I'm using a lot of structs and classes from the XNA framework.
[Edited by - AntiGuy on January 25, 2009 4:19:20 AM]
Author
Just to make sure it wasn't something I was doing I did the following.
The output was 28.6 MB and it took about 16 seconds to save and 16 seconds to load.
To compare times I loaded a 49MB level in UnrealEd 3.0. That took around 4 seconds. It can't really be 4x slower on something that doesn't even have complex data.
if (keyAPressed) { Data.Save(new byte[1000, 1000,30], "example"); } if (keyBPressed) { Data.Load("example"); }The output was 28.6 MB and it took about 16 seconds to save and 16 seconds to load.
To compare times I loaded a 49MB level in UnrealEd 3.0. That took around 4 seconds. It can't really be 4x slower on something that doesn't even have complex data.
Almost 29MB for no data? I threw together the following:
The write result was 281 milliseconds and a 523K file. Read result was 484 milliseconds.
public class Data { private List<Vector3> test; private Random _random; public List<Vector3> Test { get { return test; } set { test = value; } } public Data() { test = new List<Vector3>(); _random = new Random(); for (int i = 0; i < 50000; i++) test.Add(new Vector3(_random.Next(0, byte.MaxValue), _random.Next(0, byte.MaxValue), _random.Next(0, byte.MaxValue))); } } public class Game1 : Microsoft.Xna.Framework.Game { GraphicsDeviceManager graphics; SpriteBatch spriteBatch; private KeyboardState _lastState; private Data _data; public Game1() { graphics = new GraphicsDeviceManager(this); Content.RootDirectory = "Content"; } protected override void Initialize() { _data = new Data(); base.Initialize(); } protected override void Update(GameTime gameTime) { KeyboardState state = Keyboard.GetState(); long start, end; if (state.IsKeyDown(Keys.A) && _lastState.IsKeyUp(Keys.A)) { start = DateTime.Now.Ticks; XmlWriterSettings settings = new XmlWriterSettings(); settings.Indent = true; XmlWriter writer; writer = XmlWriter.Create("data.xml", settings); IntermediateSerializer.Serialize(writer, _data, null); writer.Close(); end = DateTime.Now.Ticks; TimeSpan elapsedSpan = new TimeSpan(end-start); Console.WriteLine(elapsedSpan.Milliseconds.ToString()); } else if (state.IsKeyDown(Keys.B) && _lastState.IsKeyUp(Keys.B)) { start = DateTime.Now.Ticks; _data = LoadData(); end = DateTime.Now.Ticks; TimeSpan elapsedSpan = new TimeSpan(end - start); Console.WriteLine(elapsedSpan.Milliseconds.ToString()); } _lastState = state; base.Update(gameTime); } private Data LoadData() { try { XmlReader reader = XmlReader.Create(new FileStream("data.xml", FileMode.Open)); Data data = IntermediateSerializer.Deserialize<Data>(reader, null); reader.Close(); return data; } catch (Exception ex) { return null; } } }The write result was 281 milliseconds and a 523K file. Read result was 484 milliseconds.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
