Jump to content

  • Log In with Google      Sign In   
  • Create Account

Banner advertising on our site currently available from just $5!


1. Learn about the promo. 2. Sign up for GDNet+. 3. Set up your advert!


EDI

Member Since 05 Jan 2002
Offline Last Active Yesterday, 09:34 PM

Posts I've Made

In Topic: Tiny Serialization Library

22 June 2015 - 09:38 AM

I'm not entirely sure how well this will work for all cases. And it only sort of works. There are errors in it, and I hadn't been too bothered to fix it.

It's a very butchered version of this post here.
http://www.randygaul.net/2013/01/05/c-reflection-part-5-automated-serialization/

Or.... well inspired from...

The requirement is some form of meta-data system or an RTTI built into your engine.

 

My method of serialization works like this.

Each item that can be serialized has an int that's very similar to the Radix search used in Stateless rendering.

 

|System|Children|Parent|Order|Type| // There is an error here... I've hadn't quite figured out Radix yet.

Naturally Type and Order can just be #Define flags.

 

 

Because these items are all looped upon in my game engine. I simply have the system wait till it's time to save before "registering"
When it's time to register. Items flagged with Serialize are copied into a buffer, and quickly ordered. Most data is saved as Entities and Components.

as it streams out bytedata, Everything is "carefully" ordered.

And... here's where things are a little wonky. I hadn't quite gotten it to the point to work fully on reload. It's capable of Loading up Entities... but components aren't loaded correctly.

 

Note... most of the code was written late at night and partially drunk. So all of this could just literally be unnecessary or overly complicated.

 

EDIT: Reading back over the code... I don't think it's worth the effort of implementing.

*nod* ...sounds like your method needs a bit more thinking; but at first glance it doesn't seem to be generalized; or allow references/cycles, or support member addition/drop-out.


In Topic: Tiny Serialization Library

19 June 2015 - 09:06 AM

 

definitely no attention is yet paid to hardening it against deliberate... attack of sorts; the key is to first iron out the features

The reality is that you can't design an API and then bolt security on to it - security and reliability have to be primary design goals.

As a trivial example, consider the millions of insecure programs running around because the gets() function doesn't take a size argument. The API and semantics of fgets() are fundamentally different to solve that, and we still can't take the broken version out of circulation due to all the legacy crap that depends on it...

 

 

I've no intention of, 'bolting' security on; I'll refactor where neccesary and pass on whatever unfortunet realities can't be hidden (as in differing function calls and arguments)

 

 

 

I dunno, I'm skimming through your code and it seems like serializing objects in yours is a crapload of work and code to write, plus you have to maintain matching read/write functions. Is that an accurate assessment?

A crapload of work, is of course subjective; you definitely have to maintain read/write functions; though out of ordering and matching is up to you by design.

I'd argue that having to maintain a parsable 'def' file of sorts for protobuf or the like is roughly as bad; but I am looking at adding some options which can allow for a single 'serialize' function for 90% of use cases where you wont care about having deliberate control over reads.

 

Well, this is what I did. It's kinda hard coded to JSON but it could be patched up for multiple backends easily enough.

 

Thanks for the code share Promit; that is roughly what I am invisioning, it certainly seems like many cases can be covered by a single method.


In Topic: Tiny Serialization Library

18 June 2015 - 07:15 PM

I dunno, I'm skimming through your code and it seems like serializing objects in yours is a crapload of work and code to write, plus you have to maintain matching read/write functions. Is that an accurate assessment?

A crapload of work, is of course subjective; you definitely have to maintain read/write functions; though out of ordering and matching is up to you by design.

I'd argue that having to maintain a parsable 'def' file of sorts for protobuf or the like is roughly as bad; but I am looking at adding some options which can allow for a single 'serialize' function for 90% of use cases where you wont care about having deliberate control over reads.

 

Most of the low level I/O, if it fails, returns uninitialize data at best (which makes debugging nice and nondeterministic.)

There's no sane bounds checking - it appears trivial to create a tiny file which will eat all your memory when using read_string.

If you're lucky, it will throw std::bad_alloc or dereference null (depending on if exceptions are available or not).  If you're unlucky you'll exhaust all available memory and have OOM crashes later elsewhere.

 

I wouldn't consider this usable even in a non-hostile environment currently - savegame corruption happens, I don't want it crashing my titles on startup.

 

In a hostile environment, your attacker will use l=0xFFFFFFFF, the call to new char[0xFFFFFFFF+1] will succeed - it's the same as new char[0] which returns a unique non-null pointer.  The resulting read() call *without* the +1 will then be the start of the buffer overflow, likely probed for possible use in code injection attacks...

 

Throw SDL MiniFuzz or other fuzzing tools at this if you want to harden it up...

Thanks MM; definitely no attention is yet paid to hardening it against deliberate... attack of sorts; the key is to first iron our the features; will definitely try a fuzzing tool when it comes time to make the readers bulletproof.


In Topic: Tiny Serialization Library

17 June 2015 - 08:20 PM

Zipster is correct, variable 'versioning' conditional blocks allow you to read a variable in an old type, and convert/promote it to a new type.

In the case of OMS you can 'peek_type'  when reading back a property if your design suspects a change; this is (in my experience) rare.

The majority of cases are adding new properties and dropping old ones; which should be simple as possible IMO.


In Topic: Tiny Serialization Library

17 June 2015 - 08:16 AM

 


I looked into google protocol buffers as well, and the code-pre-processor/generation feels like an awful hack to me.

It may not suit your particular needs, but for most users, protocol buffer's code-generation is really its #1 strength.

 

Picture a fairly typical modern-day web service: you have a Java web server, a REST client written in JavaScript, a native client written in Objective-C on iOS, and maybe an acceptance test suite written in Python. Would you really want to implement (and test!) the same serialisation code in 4 different languages? That would be an unmaintainable nightmare.

 

And thus we have tools like Protobuf to let us write our specifications just once, and never again deal with this sort of compatibility problem.

 

I suppose code generation with external tools is OK for some folks; in my case I've never wanted to use a code-generation technique.

Protobuf afaik also does not support object references/cycles; without building your own system atop it; which is a large part of the compexity to begin with.   wire-formats like XML, JSON suffer the same issues; but, references and cycles are extremely common occurences in game state formats.

 

 

it was huge, and complicated; and also did not have important features such as; adding/removing serialized members; without having to write version conditionals.

Metaclass management is not exactly simple or good usage of memory resources. The reason version conditionals are used often is because the solution takes almost no extra code to do right and supports the vast majority of cases (variable addition and removal) without any extra effort.

In my own rather simple, template-based library, I just have to add a few arguments to a function call, like so:
arch( size, arch.version >= 4, 0 ); // variable to serialize, condition, default value 
Does not seem that hard. Besides, I use the same function for serialization and unserialization and it works across two different serializers (text and binary).

and the code-pre-processor/generation feels like an awful hack to me.

Not sure where you got that idea. Code generation has been around to solve various metaprogramming issues for a long time. Whoever makes you think it's a hack, better not take such "advice". Any method has its good sides and its bad sides and evaluating it by "feels" would be a big mistake. Try it, improve it, change it for something different. Exhaust all options to have a better idea of what works and what doesn't. It really is the only way.

As for your solution, what you're proposing is writing huge read/write functions that require lots of code, lots of declarations and lots of void pointer workarounds. Furthermore, it appears that only iostream is supported even though it would take just a two-function interface to read from and write to just about anything. Quite frankly, I don't see the point of using dynamic memory for what could easily reside in static memory as well.

 

First I'll address version blocks; in any /real/ complex game, especially where change is the norm; with version blocks you will accure 'entropy'; having to specify version blocks for new members and member drop-outs is a gross misuse of technical resources; and unescapably creates technical debt; that you will live with, until you decide to make a clean format break (breaking past user saves preceeding that point).

Concerning code generation; a large degree boils down to preference; I'll admit. If you're ok with preparing special markup and running external tools, when you essentially don't have to, that is obviously an option.

Concerning iostream, it is an obvious choice, as it supports both file level and in-memory (stringstream) for deep copying etc.

The issue with the 'two function' solution for arbitrary readers/writers is that for naive iostream usage, you end up having to tie it in.
You could say implementing your own format against iostream is a solution as well.


PARTNERS