So I am still on my long journey to create a 2D tile based side scroller. Today I jumped into the part where I save my world to the harddisk instead of keeping it in the memory.
Currently what I do is, serialize the 2D Tile array that I've made. The size I test on is 10240 * 512. Now I know I can optimize the file size further, but currently this gives me a file size of roughly 305MB.(I am planning to have worlds much bigger than this one) My problem occurs when I try to deserialize the array. The game freezes, and rapidly swallows more and more memory, and after a while (5-10 mins) it will throw an out of memory exception. Is this due to the huge size of the file? Or are there other factors I should consider. My code for serializing and deserializing is here:


public void SerializeWorld(string fileName)
{
using (Stream stream = File.Open(fileName, FileMode.Create))
{
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(stream, world);
}
}
public static void DeserializeWorld(string filename)
{
using (Stream stream = File.Open(filename, FileMode.Open))
{
BinaryFormatter bf = new BinaryFormatter();
world = (Tile[,])bf.Deserialize(stream);
}
}


In the above code world is the array where I keep my tiles. I've tested this in two ways:
On both tests I create the world as I usually do, and then serialize it into a file right after the creation. The difference is when I load it. First I tried loading it while the game was running, to see if It would be able to load the world without having to leave the game session. Failing this I tried simply loading the world when I start the game, but with the same results- Game freezes for 5-10 mins, swallows memory and ends up with Out of memory exception.

Now the way I believe I should do this is divide the world into chunks, and only keep 9 chunks loaded at any one time. (Normally it would be 3, but I need to consider up and down as well). This would also require me to keep each chunk in its own file, giving me tons of smaller files. This is sort of okay with me as long as this does not cause any other issues? I would then load those 9 chunks into my in memory world array, and release the ones out of range. Currently I'm not quite sure how to do this effectively.
Maybe keeping it in small files, might fix the problem with deserialization?

So to sum it up:
- Am I right to assume that the 305MB file, is the cause of the freeze and out of memory exception when I deserialize?
- How should I correctly handle chunks?
- Will it be an issue to have multiple smaller files, where each file represents one chunk?
- Are there anything wrong with the way I de- and serialize right now, and should I consider other methods?

Thank you for reading.

The following is the current state of my Tile class.

[Serializable]
/// <summary>
/// Stores the appearance and collision behavior of a tile.
/// </summary>
public class Tile
{
public TileType Type;
public TileType BackgroundType;
public TileOrientation Orientation;
protected short _directLightLevel;
protected short _ambientLightLevel;
protected ushort _flickerTimer;
protected bool _lightCalculated;
protected byte _variant;
/// <summary>
/// Constructs a new tile.
/// </summary>
public Tile(TileType type, TileType backgroundType)
{
Type = type;
BackgroundType = backgroundType;
Orientation = TileOrientation.None;
_variant = 0;
_lightCalculated = false;
if (type == TileType.None)
{
if (backgroundType == TileType.None)
_ambientLightLevel = Light.AmbientLight;
else
_ambientLightLevel = 0;
_directLightLevel = 0;
_flickerTimer = 0;
}
else if (type == TileType.Torch)
{
_directLightLevel = 255;
_ambientLightLevel = 0;
_flickerTimer = 1;
}
else
{
_directLightLevel = 0;
_ambientLightLevel = 0;
_flickerTimer = 0;
}
}
public virtual TileCollision Collision
{
get { return Terrain.TerrainGenerator.tileInfo[(int)Type].Collision; }
}
public virtual ToolType Tool
{
get { return Terrain.TerrainGenerator.tileInfo[(int)Type].Tool; }
}
public byte Variant
{
get { return _variant; }
set { _variant = value; }
}
public bool LightCalculated
{
get { return _lightCalculated; }
set { _lightCalculated = value; }
}
public short AmbientLightLevel
{
get { return _ambientLightLevel; }
set { _ambientLightLevel = value; }
}
public short DirectLightLevel
{
get { return _directLightLevel; }
set { _directLightLevel = value; }
}
public short LightLevel
{
get
{
if (_directLightLevel + _ambientLightLevel > 255)
return 255;
if (_directLightLevel + _ambientLightLevel < 0)
return 0;
return (short)(_directLightLevel + _ambientLightLevel);
}
}
public ushort FlickerTimer
{
get { return _flickerTimer; }
set { _flickerTimer = value; }
}
}


I recently removed a lot of data from the tile class, and added it to a static tile class, where it keeps information based on type etc. I have already planned to remove the enumerators in the tile and replace them with bytes, I figure there's a lot more room to spare there.
I do have a specific type of Tile that inherits from this one, I call Dynamic tile. That is for tiles that span over more than one tile, and that have changing states (besides orientation).
If you have any suggestions to optimize the class above, I am all ears - And a question for that one - Would a public variable take less memory than accessing it through a property? As you might notice I've experimented with that, but no visible results.

I will check out the link you provided and see if there are things I've overlooked there.
Thanks for your reply

woow, >10000 tiles in x. What is the size of tile. Is level not too long. For example if tile 64X64 pixels and screen is 1280px and lets take one screen last about 5 seconds (normal scrolling speed) to fly throu it will be 64*10240*5/1280/60=42 minutes. Not too log for level?

If tiles are in most cases the same or and they have rules of connect with each other , so it could be very procedural. Tiles could be generated proceduraly.

I have already planned to remove the enumerators in the tile and replace them with bytes

You can make enums be byte-sized by doing the following:

enum TileType : byte
{
Grass,
Dirt,
// etc....
}

As for serialization, you should be using custom serialization for something like this. I don't know the details of how .net serializes classes, but I'm sure it needs a bunch of metadata along with the actual data. It may be convenient, but it doesn't seem like the right tool for a scenario like this.

It also looks like you have a bunch of runtime state in your Tile class (_flickerTimer, _lightCalculated, etc...), in addition to "static" state (_BackgroundType, etc..). You can probably avoid serializing that.

Assuming the only important "static" information is BackgroundType, Orientation, _directionLightLevel and _ambientLightLevel, then you've got 6 bytes, for a total of around 30MB for your 10240 * 512 level. Looking at your code, it may be that _directionLightLevel and _ambientLightLevel can be inferred from the tile type? If so, then you really only need 2 bytes per tile, so 10MB.

You could probably substantially reduce this with some compression algorithm, since I assume there are a lot of tiles of similar types next to each other.

That was a lot of good responses I got there

woow, >10000 tiles in x. What is the size of tile. Is level not too long. For example if tile 64X64 pixels and screen is 1280px and lets take one screen last about 5 seconds (normal scrolling speed) to fly throu it will be 64*10240*5/1280/60=42 minutes. Not too log for level?

I am aiming at very large worlds like in Terraria, so the size I present is rather small. (I am doing 16X16 tiles though)

Chunks is indeed the way to go for such a large world, and having them in many small files should not be a problem at all. Then you can begin to stream new chunks directly from the files as the player gets close to them.
As for that last question: Properties makes no significant difference to the size of the class, so use them as much as you want.

When doing chunks, do you then Deserialize small amounts on the run? Or how is it handled?
And thanks for the info on properties, just the answer I was looking for.

If that did not solve your problem, try this: http://www.downloadmoreram.com

Good one lol - But I'm pretty sure I don't need more RAM, 8GB should be plenty for this - I will try out your suggestions.

I wonder whether memory allocation may be particularly inefficient if you're serialising a generic List or similar, e.g. hard for it to know up front what memory to allocate.

I serialize an array with a pre-determined size.

You can make enums be byte-sized by doing the following:

enum TileType : byte
{
Grass,
Dirt,
// etc....
}

As for serialization, you should be using custom serialization for something like this. I don't know the details of how .net serializes classes, but I'm sure it needs a bunch of metadata along with the actual data. It may be convenient, but it doesn't seem like the right tool for a scenario like this.

It also looks like you have a bunch of runtime state in your Tile class (_flickerTimer, _lightCalculated, etc...), in addition to "static" state (_BackgroundType, etc..). You can probably avoid serializing that.

Assuming the only important "static" information is BackgroundType, Orientation, _directionLightLevel and _ambientLightLevel, then you've got 6 bytes, for a total of around 30MB for your 10240 * 512 level. Looking at your code, it may be that _directionLightLevel and _ambientLightLevel can be inferred from the tile type? If so, then you really only need 2 bytes per tile, so 10MB.

You could probably substantially reduce this with some compression algorithm, since I assume there are a lot of tiles of similar types next to each other.

I will definetly look into this- I did not know that you could do that to enumerators, thanks a bunch for that one.

All in all it seems I've underestimated the entire topic of serialization. I am very grateful for all the responses you've given me. I think I will try to reduce the size of the tiles more, and look more into custom serialization.
Right from the beginning of this project, the two things I've "feared" the most is this, and random terrain generation.