Jump to content

  • Log In with Google      Sign In   
  • Create Account

#ActualNypyren

Posted 04 March 2013 - 10:47 PM

For structs, there are generally two problems:

- You add a new field, but the old file doesn't have it.
- You remove a field, but the old file still has it.

There are two main types of implementations I've seen:

=== Option 1 ===

Keep a version number for each struct, serialize the version number (either per struct or once for the entire save file), then write a deserializer which can handle any version up to now. The main benefit here is that you can fread the entire struct if its version number matches your code's struct. If not, you fall back to a generalized reader which if/else reads each individual field based on whether it existed at that version or not (including skipping fields that you've removed). The idea is that it only uses the slow deserializer when the app gets patched, and then when you overwrite the save file, it will be blazing fast again.

Terraria uses the if/else approach, but most of its loading time is spent rebuilding data that isn't actually saved to the file.

The code can look like:
if (version == kCurrentVersion)
{
  ReadEntireStruct(); // fread in C, other languages may or may not have a fast equivalent.
}
else
{
  field1 = ReadString();

  if (version > 12)
    field2 = ReadInt32();

  if (version > 13) // Needed to increase bitfield size to 32 bits from 16
    field3 = ReadInt32();
  else
    field3 = ReadInt16();
}
=== OPTION 2 ===

Each field keeps a 'field id', and the serialized struct stores that field ID before the field contents. In addition, if you want the ability to deserialize "unknown" (removed) fields, you need to add more data that indicates how long (or what type) the field is.

Google's protobuf uses this approach.

The code here can be written in various ways. For example, protobuf.net can generate deserialization classes on the fly and compile them with JIT.
while (fieldInfo = ReadFieldId())
{
  switch (fieldInfo.id)
  {
    case 1: field1 = ReadString(); break;
    case 2: field2 = ReadInt32(); break;
    case 3: field3 = ReadInt16(); break;
    case 4: field3 = ReadInt32(); break;   // notice the field id also had to change when changing the field's type.

    default: SkipField(); break;
  }
}

#6Nypyren

Posted 04 March 2013 - 10:36 PM

For structs, there are generally two problems:

- You add a new field, but the old file doesn't have it.
- You remove a field, but the old file still has it.

There are two main types of implementations I've seen:

=== Option 1 ===

Keep a version number for each struct, serialize the version number (either per struct or once for the entire save file), then write a deserializer which can handle any format. The main benefit here is that you can fread the entire struct if its version number matches your code's struct. If not, you fall back to a generalized reader which if/else reads each individual field based on whether it existed at that version or not (including skipping fields that you've removed). The idea is that it only uses the slow deserializer when the app gets patched, and then when you overwrite the save file, it will be blazing fast again.

Terraria uses the if/else approach, but most of its loading time is spent rebuilding data that isn't actually saved to the file.

The code can look like:
if (version == kCurrentVersion)
{
  ReadEntireStruct(); // fread in C, other languages may or may not have a fast equivalent.
}
else
{
  field1 = ReadString();

  if (version > 12)
    field2 = ReadInt32();

  if (version > 13) // Needed to increase bitfield size to 32 bits from 16
    field3 = ReadInt32();
  else
    field3 = ReadInt16();
}
=== OPTION 2 ===

Each field keeps a 'field id', and the serialized struct stores that field ID before the field contents. In addition, if you want the ability to deserialize "unknown" (removed) fields, you need to add more data that indicates how long (or what type) the field is.

Google's protobuf uses this approach.

The code here can be written in various ways. For example, protobuf.net can generate deserialization classes on the fly and compile them with JIT.
while (fieldInfo = ReadFieldId())
{
  switch (fieldInfo.id)
  {
    case 1: field1 = ReadString(); break;
    case 2: field2 = ReadInt32(); break;
    case 3: field3 = ReadInt16(); break;
    case 4: field3 = ReadInt32(); break;   // notice the field id also had to change when changing the field's type.

    default: SkipField(); break;
  }
}

#5Nypyren

Posted 04 March 2013 - 10:36 PM

For structs, there are generally two problems:

- You add a new field, but the old file doesn't have it.
- You remove a field, but the old file still has it.

There are two main types of implementations I've seen:

- Keep a version number for each struct, serialize the version number (either per struct or once for the entire save file), then write a deserializer which can handle any format. The main benefit here is that you can fread the entire struct if its version number matches your code's struct. If not, you fall back to a generalized reader which if/else reads each individual field based on whether it existed at that version or not (including skipping fields that you've removed). The idea is that it only uses the slow deserializer when the app gets patched, and then when you overwrite the save file, it will be blazing fast again.

Terraria uses the if/else approach, but most of its loading time is spent rebuilding data that isn't actually saved to the file.

The code can look like:
if (version == kCurrentVersion)
{
  ReadEntireStruct(); // fread in C, other languages may or may not have a fast equivalent.
}
else
{
  field1 = ReadString();

  if (version > 12)
    field2 = ReadInt32();

  if (version > 13) // Needed to increase bitfield size to 32 bits from 16
    field3 = ReadInt32();
  else
    field3 = ReadInt16();
}
- Each field keeps a 'field id', and the serialized struct stores that field ID before the field contents. In addition, if you want the ability to deserialize "unknown" (removed) fields, you need to add more data that indicates how long (or what type) the field is.

Google's protobuf uses this approach.

The code here can be written in various ways. For example, protobuf.net can generate deserialization classes on the fly and compile them with JIT.
while (fieldInfo = ReadFieldId())
{
  switch (fieldInfo.id)
  {
    case 1: field1 = ReadString(); break;
    case 2: field2 = ReadInt32(); break;
    case 3: field3 = ReadInt16(); break;
    case 4: field3 = ReadInt32(); break;   // notice the field id also had to change when changing the field's type.

    default: SkipField(); break;
  }
}

#4Nypyren

Posted 04 March 2013 - 10:27 PM

For structs, there are generally two problems:

- You add a new field, but the old file doesn't have it.
- You remove a field, but the old file still has it.

There are two main types of implementations I've seen:

- Keep a version number for each struct, serialize the version number (either per struct or once for the entire save file), then write a deserializer which can handle any format. The main benefit here is that you can fread the entire struct if its version number matches your code's struct. If not, you fall back to a generalized reader which if/else reads each individual field based on whether it existed at that version or not (including skipping fields that you've removed). The idea is that it only uses the slow deserializer when the app gets patched, and then when you overwrite the save file, it will be blazing fast again.

Terraria uses the if/else approach, but most of its loading time is spent rebuilding data that isn't actually saved to the file.


- Each field keeps a 'field id', and the serialized struct stores that field ID before the field contents. In addition, if you want the ability to deserialize "unknown" (removed) fields, you need to add more data that indicates how long (or what type) the field is.

Google's protobuf uses this approach.

#3Nypyren

Posted 04 March 2013 - 10:24 PM

For structs, there are generally two problems:

- You add a new field, but the old file doesn't have it.
- You remove a field, but the old file still has it.

There are two main types of implementations I've seen:

- Keep a version number for each struct, serialize the version number (either per struct or once for the entire save file), then write a deserializer which can handle any format. The main benefit here is that you can fread the entire struct if its version number matches your code's struct. If not, you fall back to a generalized reader which if/else reads each individual field based on whether it existed at that version or not (including skipping fields that you've removed). The idea is that it only uses the slow deserializer when the app gets patched, and then when you overwrite the save file, it will be blazing fast again.

Terraria uses the if/else approach, but most of its loading time is spent rebuilding data that isn't actually saved to the file.


- Each field keeps a 'field id', and the serialized struct stores that field ID before the field contents. In addition, if you want the ability to deserialize "unknown" fields, you need to add more data that indicates how long (or what type) the field is.

Google's protobuf uses this approach.

#2Nypyren

Posted 04 March 2013 - 10:22 PM

For structs, there are generally two problems:

- You add a new field, but the old file doesn't have it.
- You remove a field, but the old file still has it.

There are two main types of implementations I've seen:

- Keep a version number for each struct, serialize the version number (either per struct or once for the entire save file), then write a deserializer which can handle any format. The main benefit here is that you can fread the entire struct if its version number matches your code's struct. If not, you fall back to a generalized reader which if/else reads each individual field based on whether it existed at that version or not (including skipping fields that you've removed). The idea is that it only uses the slow deserializer when the app gets patched, and then when you overwrite the save file, it will be blazing fast again.

Terraria uses the if/else approach.


- Each field keeps a 'field id', and the serialized struct stores that field ID before the field contents. In addition, if you want the ability to deserialize "unknown" fields, you need to add more data that indicates how long (or what type) the field is.

Google's protobuf uses this approach.

PARTNERS