Hello all!

I've been searching the internet for some time, now, on the topic, and haven't seemed to find anything of value.

The situation at hand, is that I have vectors of objects serialized, using Boost serialization. It works great, and is an effective tool. Although, the larger the files get, the longer they take to load. This is especially painful when you are only looking for a very specific piece (0.001%) of the actual file. What's worse, is that the only way to circumvent it is to have so many different files that it is ridiculous.

What I'd like to do, is, be able to iterate through any container(s) within the file, and load the specific one that I'm looking for.

For example, an object that I am looking for has an ID # of 9289. Then, be able to find the object with that ID # within the file, load it, and use it within my program.

An alternative is to be able to access a known area of the file.

For example, wanting to load the entire 54th vector within the file.

Could somebody please correct me in regards to paragraph #2 here, or lead the way to a better solution?

Thank you in advance,

Kevin

I have not used Boost serialization library yet. But I know you can specify the input/output file to store your archive.

Can you group your objects or containers before you serialize them? Then when you desterilize you can simply choose the archive you need instead of loading everything from a giant archive.

Use an index table.

1. Reserve some space at the beginning of the file for the index table.
2. Serialize each entity, noting the write position of the stream, and store that and the entity ID into the table.
3. After serializing the entities, sort the table by entity ID.
4. Write the number of entities and the contents of the table to the beginning of the file.

When looking for a specific set of entities, all you'll need to do is to read the entire index, binary search for the entity IDs you're looking for, set the read position of the stream, then deserialize the entity.