Or do you have fixed numbers of each type of game object?

bingo.

but no reason it couldn't be variable.

an example:

lets start with a character struct with just a name, x, and y. and we load and save a variable number of these. no problem, write the number of character structs, followed by the name, x, and y for each struct. very standard stuff, right? when you read, you read how many there are, then do a loop of that number of iterations reading in the values for each struct.

ok, now - add something else - how about z? it should work for this example.

so you leave the existing code alone, and add new code at the end:

// you've already written num_structs, so no need to do it twice. just write the (new) struct data.

for each struct, write z

so you're saving all the version 1 format data first (name, x and y), followed by all the version 2 data (z).

and load just reverses the process:

read num_structs

for i=0 i<num_structs i++

{

read name

read x

read y

}

// you could be clever and realize you probably don't need to write num_structs twice, so for new format 2 data just do:

for i=0 i<num_structs

{

read z

}

What if you add a property to animal? You have to write the new properties for all animals at the end of your save function?

any new variable you want to add to the format goes at the end. does that answer your question? not sure what you mean by a property.

What if then if you increase the fixed number of animals? You'd need to write the first n animals with m properties, then later on n animals with the 1 new property, then later on more animals with (m + 1) properties.

simply designing the format from the get go for a variable number of objects as in my example above precludes this issue.

if you kick up the max number of avatars in the game, a savegame will have at most the old max, and you'll be saving up to the the new max.

going the other direction, reducing max_avatars, you might have to throw away data from old file formats, if you have no place to put it / no use for it.

as for writing a fixed vs variable number of objects, i tend to use memory pools implemented using arrays for most everything for which there can be more than once instance in the game. IE just about everything. since i'm dealing with arrays, i personally find it easier to simply dump the entire player and npc memory pools to disk, rather than only writing active ones (IE a variable number of objects). to squeeze a few more millisecs out of my load and save times, i could switch to only loading and saving active ones. but many are relational databases, and memory dumps means no pointer index fixup on load to get everything related correctly again. about the only fixup i need on load is creating animation controllers for active characters.

i personally find it easier to simply dump the entire player and npc memory pools to disk, rather than only writing active ones (IE a variable number of objects). to squeeze a few more millisecs out of my load and save times, i could switch to only loading and saving active ones

You said you're saving 72MB of data in 5 seconds. I'm sure you don't have 72MB of actual save game data, so most of that must be inactive objects in memory pools. Saving only valid objects would probably not squeeze a few milliseconds off your load/save times, but instead reduce them by an order of magnitude. I/O is going to dominate your load/save times.

I/O is going to dominate your load/save times

A shitty 5,400 RPM hard drive writes at 100 MB/s these days. Any decent SSD should be able to write above 400 MB/s.

In light of those figures, taking 5 seconds to write ~75 MB means you are doing something criminally wrong.

I should clarify, I don't mean that as a personal attack on Norman Barrows - one of my professional tasks last year was optimising file I/O for an in-house engine, and one of the takeaways is that a large set (possibly a majority) of software is failing miserably at performant file I/O.

And one of the main reasons? Neglecting to think about buffering. I don't have access to Windows with it's *_nolock variants right now, but consider the following two programs, one which relies on libc to do the buffering, and one which buffers itself:

#include <stdio.h>

int main() {
	FILE *f = fopen("small.out", "wb");
	for (int i = 0; i < 18*1024*1024; ++i) {
		fwrite(&i, 1, sizeof(int), f);
	}
	fclose(f);
}

#include <stdio.h>

int main() {
	int buffer[1024*1024];
	FILE *f = fopen("large.out", "wb");
	for (int i = 0; i < 18; ++i) {
		for (int j = 0; j < 1024*1024; ++j) {
			buffer[j] = i*1024*1024 + j;
		}
		fwrite(&buffer, 1024*1024, sizeof(int), f);
	}
	fclose(f);
}

Fairly obviously in retrospect, our manual buffering is roughly an order of magnitude faster than libc (and adjusting the libc buffer size up to a meg makes surprisingly little difference):

$ time ./write_small 
real 0m1.234s
user 0m1.109s
sys 0m0.100s
$ time ./write_large 
real 0m0.214s
user 0m0.012s
sys 0m0.109s

Microsoft's *_nolock variants are likely a lot better in this regard (I'll try and test them tonight), but I'd be surprised if they get you all the way there - and even if they do, you likely have another issue (maybe too many cache misses while traversing your data, maybe a page fault or two) going on.

Edit:
As expected, the _fwrite_nolock() approach falls somewhere between the two extremes:

PS> Measure-Command {.\small.exe}
Seconds           : 1
Milliseconds      : 60
PS> Measure-Command {.\large.exe}
Seconds           : 0
Milliseconds      : 81
PS> Measure-Command {.\nolock.exe}
Seconds           : 0
Milliseconds      : 353

To answer the original questions....

an excellent summary.

And one of the main reasons? Neglecting to think about buffering.

it just occurred to me that buffering could be used with the method i described. the data from each format version could be buffered before writing. of course you still have to populate the write buffers and parse the read buffers. but those are ram operations, not i./o.