Disclaimer: I am writing my own thing, so this does not pertain to UE, Unity or any other professional engine.

I'm taking a critical look at my actor code and trying to figure out how best to serialize/deserialize the data for disk and network streaming. The problem right now is that there are inherently a lot of allocations happening for any given actor.

Initializing the "RPG" component of a pawn actor alone necessitates a slew of allocations as there are quite a few subcomponents that have a metric crapton of fields/arrays that do not have fixed sizes. I'd like to not only reduce load times (which in all fairness aren't an issue at this point), but also memory fragmentation as I haven't really implemented any manner of a compactification in my memory manager.

What are the common approaches and things to streamline in this case, and - more importantly - are they worth it?

The most obvious ones I can think of are:

1) convert dynamic strings to fixed-length char buffers
2) convert any vector-style arrays to fixed-size arrays
3) use field packing and relative offset pointers to coalesce several small chunks of data into a fixed-sized pool (kind of the way uniform buffers work)
4) resolve internal pointers of referenced objects to indexes into an array or hashes into a hashmap (this probably requires some kind of a an additional solution to cache the actual pointers)

The issue here is that I'm working with abstract objects that can be extended fairly arbitrarily. That being said, while I feel reserved about capping objects containing arrays stored within an array stored within an array, I don't really see the harm in setting some fixed size/length caps on certain variables at the expense of storage. My memory allocator is generously aligned anyway, so there's already some inherent waste.

The reason I'm asking is because I'm at a point where I don't have a real-world use case in my hands and I can't really tell if worrying (or worrying to what extend) about this is worth it. Rewriting this stuff is a lot of work, but I'd rather do it before I compound the issue further. And also to have the infrastructure in place as I'd like to write some form of serialization solution in the near future.

When it comes to performance there is a simple rule:

** MEASURE EVERYTHING **

What is slow for serialization in your game may be extremely fast for serialization in my game.

In general terms data conversions and processing slow down serialization. Being able to use the data directly from disk is generally fastest, but not always. Part of that is why games use S3 texture formats (also called DXTx or DDS formats) because they can be loaded direct to the card, instead of formats like jpeg or gif or png that need to be decoded, expanded, and then sent to the card.

However, just because it makes sense in general does not mean it makes sense for your data. Sometimes compressed data is faster than raw data since there is less (slow) disk rates and more (fast) memory transfer rates. Sometimes more complex algorithms to memory-map elements into their final location and use an offset table to access everything is actually more overhead than just loading data the stupid direct way.

Many times the things you think are slow aren't actually slow, and the things you think are fast aren't actually fast. That is why you need to measure for your specific situation, then address the issues based on measurements. Make a change, and measure again to ensure the changes helped.

I am aware of the basic rule of premature optimization :).

Hence, like I mentioned on more than one occasion, my concern isn't so much about speed as it is about practicality and memory fragmentation. I'm looking for ideas, possible input on how a commercial engine might do this and trying to figure out a middle ground, which would allow me to serialize my data without too much hassle while not bending over backwards to then deserialize it.

In this case versioning my code so that I can compare microseconds is the last thing on my mind (okay, maybe not the last), because of the amount of work involved. I don't just want to dig into it and start profiling - instead, I want to write good architecture that doesn't need to be redesigned in a couple of months.

I'd argue it's a bit strange to worry about memory fragmentation, while possibly arguing for fixed-length arrays (where most of the space is unused) and talking about how your memory allocator has 'generous' alignment. Aren't you just moving the fragmentation around?

Personally, I don't recommend trying to make everything into flat data structures for fread/fwrite - I've spent literally weeks debugging problems that arose from code that did this, but which failed to anticipate some future change (to the data structure, to alignment, or padding, or the shift from 32 to 64 bits), and silently trashed or lost some data as a result. But, I know some people will disagree and say that it's worth the effort. Still, the big problem for me is that when you change your format, all your data is dead. A careful versioning system can help.

One thing that is worth doing, if practical, is replacing dynamic length strings with string IDs, hashes, or indices. Unreal has this via the FName concept. Your structures that contain strings end up containing small fixed-sized IDs/hashes/indices, while the actual text lives in a hash table somewhere, and can be stored quite efficiently as one long contiguous string at the end of a file. Sometimes the actual text doesn't need to be loaded at all (e.g. if you use hashes, and just check for equality). Different approaches have different pros and cons.

In that case, your four items are the best options I know of for the general case.

#1 and #2 avoid allocations and reduce memory time. #3 and #4 (often done with an offset table) can help minimize disk time. Do all four and you reduce both disk and memory time. The more packing you do up front the more fragile the system is to modification and extension, so it is a balance.

You mention network, so the extremely high overhead of data transfer is well worth additional processing. Memory fragmentation will be specific to the game; level-based games where EVERYTHING can be loaded at once and EVERYTHING can be discarded at once fit the model of a large bundle, but they aren't particularly extensible. Longer-running systems need more handling of object lifetimes and object pools. While there is a ton of literature on various scenarios, they still boil down to special cases the options you listed: minimize allocations and transforms, minimize expensive I/O.

'd argue it's a bit strange to worry about memory fragmentation, while possibly arguing for fixed-length arrays (where most of the space is unused) and talking about how your memory allocator has 'generous' alignment. Aren't you just moving the fragmentation around?

I'm aligning allocations to 16 bytes by default. Overall that's a whole lot of wasted space. That's what I meant by generous.

What I'm arguing for here is "clumping" allocations more compactly by coalescing as many of them as possible. The only place where this matters presently in practice is when loading a new level. I'm sorta structuring stuff with world streaming in mind, though, so that's kind of the bigger general concern on my mind.

Personally, I don't recommend trying to make everything into flat data structures for fread/fwrite - I've spent literally weeks debugging problems that arose from code that did this, but which failed to anticipate some future change (to the data structure, to alignment, or padding, or the shift from 32 to 64 bits), and silently trashed or lost some data as a result. But, I know some people will disagree and say that it's worth the effort. Still, the big problem for me is that when you change your format, all your data is dead. A careful versioning system can help.

Good points all around. I was thinking of sidestepping architecture/compiler-specific stuff (32/64-bit, struct member ordering and endianess specifics) by writing a strongly typed serializer that breaks everything down to basic types and fills in each field individually. So while the data is loaded from disk in one big chunk, the deserializer still walks through each field individually and does any necessary conversions. This SHOULD avoid data loss on format change.

Basically do something like this:

struct mytype_t { 
   rect_t field1;
   int32 field2;
   int16 field3;
};
IDataSerializer ds;
mytype_t mt;
// do runtime reflection that determines field order and sets up offsets/sizes/types
DEF_SERIALIZED_TYPE(ds, mytype_t, field1, field2, field);
DEF_SERIALIZED_TYPE(ds, rect_t, lft, top, rgt, btm);

// do the same for basic types - int32, int16, etc

// use the definition above to write the object tree to disk, breaking everything down to basic types
ds.serialize(stream, mt);
// if order of fields is not changed, fill in the entire structure one basic type member at a time
ds.deserialize(stream, mt);

I couldn't really describe how I would implement versioning in a robust way right off the bat.

One thing that is worth doing, if practical, is replacing dynamic length strings with string IDs, hashes, or indices. Unreal has this via the FName concept. Your structures that contain strings end up containing small fixed-sized IDs/hashes/indices, while the actual text lives in a hash table somewhere, and can be stored quite efficiently as one long contiguous string at the end of a file. Sometimes the actual text doesn't need to be loaded at all (e.g. if you use hashes, and just check for equality). Different approaches have different pros and cons.

Indeed, many of my strings have a hash and I'v gone to some lengths to avoid data duplication by only storing a HASH-type variable where needed.

You mention network, so the extremely high overhead of data transfer is well worth additional processing.

This is a very good point. I did mention networking somewhat prematurely, though, as I don't really have any real multiplayer at this time. Worth keeping in mind, though.

You say, "the deserializer still walks through each field individually and does any necessary conversions"

Frob says, "data conversions and processing slow down serialization".

Sounds like you're trading away one problem only to get another. Without having measured anything yet, I don't think you're in a position to make that decision either way.

There is also no decent way to do versioning for arbitrarily-nested objects. You can do it at the level of individual struct types where those structs contain basic primitives, but that's about it.

This is a follow-up post after a few days of mulling things over. You guys brought up several crucial things, which I tried to take into consideration. Thanks!

Anyway, I took some time to go over my engine code and assess where and how much it would be feasible to start streamlining stuff. Presently I settled my approach to memory allocations with what Hodgman said - not worrying so much about allocations themselves so much as making sure nearby allocations retain similar lifetimes. One of the primary things I was worried about was an I/O mechanism, so I started by automating the serialization code. I haven't been able to do any profiling, because a) I don't have an actual real world dataset to work with yet and b) for now I'm neck deep in debug mode anyway. That being said, I'm using std::is_standard_layout<> to automatically coalesce trivial fields and bin whole arrays of POD-like objects. A lot more preparatory work can likely be done during compile time and I'm hoping to offload some additional infrastructure to the compiler when I finally upgrade to a one that supports constexpr.

As for runtime, using a few of my actual data structures as a baseline, simple field reordering to prioritize grouping of simple/POD-like members and taking of a few steps back to critically assess how I store the stuff in the first place yields results where I can coalesce about 60-80% of data that is read from disk into a single mem copy operation. This does assume some consistency across platforms, though: my current solution to versioning would be to store hashes of object types (which I'm thinking of calculating as a CRC of one long string (which has been stripped of whitespaces) obtained by concatenating all field types of serialized object) in the data file's header. At this point I can still decide to read data as-is, even if the hashes don't match to partially load an object, or - as suggested above - rebuild the dataset.

This does assume some consistency across platforms, though

Yeah, that's exactly what wasted our team about a month of coder time trying to fix up. Not only do you have to consider type sizes, but you have to consider alignment, and padding. And changing compiler defaults. And new platforms.

Take a simple struct of `int a; char b; vector3 c;`. This will give you a single unambiguous CRC hash. But, given any arbitrary platform or compiler setting, do you really know what sizeof(a), offsetof(b), or offsetof( c ) are?

You can smash some of this with #pragma pack. Just hope you don't have any objects that have picky alignment requirements.