Jump to content

  • Log In with Google      Sign In   
  • Create Account


Member Since 08 Mar 2000
Offline Last Active Today, 10:31 AM

#5307851 Low level serialisation strategies

Posted by on 25 August 2016 - 09:02 AM

There is no reading code -- you declare those structures and then you use them without a deserialization step.


Right, I guess I just can't mentally work out how that operates given the code that you show, because (for example) the writing code has appeared to add a stringtable at the end that didn't exist prior to serialisation. No doubt there's something tied up in the StringOffset and Offset types that does the magic!


Some code I've worked with was exactly like that - per-field serialization, load-blob-and-cast deserialization. And I think that was the approach that caused me the most problems, mostly because there was no real attempt at making the types work well with it, just a mixture of POD types (which work fine, when you stick to explicitly-sized types), structs (which align differently on different platforms, breaking stuff), and pointers (which change size based on architecture, breaking stuff, and requiring a fix-up stage, and some sort of composition vs. association strategy). I expect that can be mitigated a lot if the pointers are all replaced with smart pointers (and ideally ones that understand association vs composition).


Personally I wouldn't really want to change a lot of the types involved in the data just to facilitate quicker serialisation; but I can see that it's definitely something people can make a case for (and especially for the GPU data like you said).

#5307840 Converting world rotation to body rotation

Posted by on 25 August 2016 - 07:40 AM

You can't know anything intrinsically about the plane orientation just from the camera orientation alone. If there is another data source you can use to get the plane orientation, you can just compare the two.

#5307837 Low level serialisation strategies

Posted by on 25 August 2016 - 07:21 AM

You only use this for plain old data. If you're building an engine system and want to be able to load it form disc with a memcpy, then you're able to make the choice to design it as POD.


I guess I find it hard to imagine a situation where I could do that and still find the code convenient to work with. Some objects are intrinsically hierarchical, and some have varying length contents; trying to flatten all that just seems to move the complexity out of the serialisation and into everywhere else in the engine.




your build system needs to generate binaries for each platform


Sure, but it's easier said than done, especially when you have pointers and you're going for a memcpy/fwrite type of approach.


Re: the stringoffset stuff - what I understand of it looks much like some of the pointer serialisation stuff I've dealt with in the past, but the code is essentially back into writing out each field one by one, right? Which isn't a bad thing, just that I thought you were trying to avoid that. And the reading code would appear to have to do the same.

#5307823 Low level serialisation strategies

Posted by on 25 August 2016 - 05:49 AM

I just finished writing an article on this exact topic - its currently pending approval - hopefully some of the members interested in this topic will participate in the peer review!


I just skimmed it - what you have there is fairly close to what I would use myself. The pros are that it's all explicit; the cons are that you're having to serialise each field individually, and it doesn't really do anything to handle pointers or weak references, which is a big part of fast serialisation for toolchains. (e.g. The many-to-many relationship between models and textures - you don't want each model to write out its own copy of each texture.)

#5307822 The right tools for making a FM-inspired database reliant manager game?

Posted by on 25 August 2016 - 05:35 AM

For a game like that, I think the hardest bit is picking a tool that can render the menus and spreadsheets. Unity is bad at it. Most game engines are bad at it. I'd be tempted to make it in HTML5 because that's almost the only game in town when it comes to making a usable UI with automatic layout capability.


How you handle the database is up to you, but you can easily use Sqlite for such a task if you're familiar with relational DBs.

#5307816 Simple Data Structure for quick look-up of "closeness" on 2D grid?

Posted by on 25 August 2016 - 04:35 AM

Make a coarse grid, with squares of width equal to the radius of interest. For each object X, you only ever need to examine the grid square that includes the object or the 8 that border on it. Done.


Theoretically there are weird and wonderful algorithms that would allow you change this 2D problem into a 1D problem, involving space-filling curves, but I can almost guarantee you that the cost of creating and maintaining the structure will far exceed any benefit you get from it.

#5307815 Low level serialisation strategies

Posted by on 25 August 2016 - 03:45 AM

Often we don't even fix up pointers on load. I've a template library containing pointer-as-relative-offset, pointer-as-absolute-offset, fixed array, fixed hash table, fixed string (with optional length prepended to the chars, and the hash appearing next to the offset to the chars), etc. Instead of doing a pass over the data to deserialize these, they're just left as is, and pointers are computed on the fly in operator->, etc. This can be a big win where you know a 2byte offset is enough, but a pointer would be 8bytes.

This sounds fascinating but I have no idea how that would work. Say you have this object:

class Xyz
    std::string name;
    u32 value1;

What does the writing code look like? Do you write each field out individually, telling it that you want 'name' to be written as a fixed string? Because I'm encountering serialisation strategies that basically mandate removal of that string from the class, replacement with a string hash or some other POD type, then an fwrite (or equivalent).


(Or, worse, I've seen a system that expected to read objects back in a single chunk, but required each field to be written individually, so that it could mangle pointers on the way out. This kind of gives you the worst of both worlds. But hey, fast loads and you can still use pointers!)

#5307813 Low level serialisation strategies

Posted by on 25 August 2016 - 03:37 AM

...a full serialisation system (e.g. with each field getting read or written individually)...

Never found a need for this personally. If you have multiple fields that need to be read/written, what's the argument against doing it in a single pass?

1. Because 99% of the objects in the engines I've worked with are not Plain Old Data
2. Because the data is padded or aligned differently on different platforms

Issue 1 I've seen approached with wacky pointer-mangling tricks. Then if one bit of data is wrong on the way in, everything's completely wrong. It seems to be complicated by requiring all the data to be coalesced into one contiguous chunk, or perhaps using several chunks each marked with their former location. Messy. I also have no idea if anyone ever got this to work on standard library objects.

Issue 2 I've seen approached with a variety of brittle attempts at manual padding, switching member ordering around, macro-d types that add padding depending on platform, etc. This seems to be a massive source of bugs because you don't always notice data that's getting splashed over the wrong fields, offset 4 bytes earlier (for example).

I can appreciate the hypothetical speed benefits of this but given how error-prone they are, I wonder whether there is any real benefit.

#5307628 Low level serialisation strategies

Posted by on 24 August 2016 - 08:56 AM

I've been surprised to see that quite a few developers are still using serialisation strategies that are equivalent to fread/fwriting structs. Sometimes they do fancy things to change pointers into offsets at save time and then fix them up at load time, but the emphasis is still on minimising memory allocations and being able to write directly into the final structure with no extra copying. As someone out in Unity/C# land for a few years, this came as a surprise to me when I got back to working with C++ code.


My main question (especially for people who've worked on shipped games) is - do you see this often? Or are you using safer (and easier to debug) methods, whether a full serialisation system (e.g. with each field getting read or written individually), or a 3rd party serialisation system like Protocol Buffers, FlatBuffers, Capt'n Proto, etc? The latter seem to have their own limitations, such as including the schema in the data being transmitted, or expecting you to use their generated data structures rather than working from your own. Are people optimising for debuggability, or deserialisation speed, or size on disk, or compatibility with 3rd party tools, or...?

#5307616 2D - Circles at high velocity become elliptical to the eye. How to workaround?

Posted by on 24 August 2016 - 08:12 AM

It could be an artifact of display latency where the response time to change a pixel is slow enough that you start to notice it. If so, you might improve the problem by reducing the contrast so that the pixels have less 'luminance distance' to travel when the sprite enters or leaves their zone.

#5307608 optimising costs when deploying mobile game+web platform

Posted by on 24 August 2016 - 07:25 AM

While it makes for a system that is somewhat easier to work with on a large scale, it requires significant effort to create so I doubt it is what you are looking for.

Oh, so true. I spent much of the last year wrestling with Amazon Elastic Beanstalk, a system that is supposed to make it easier to deploy Python web apps on Amazon Web Services (compared to doing it the 'normal' way). In truth, it's a bunch of undocumented scripts with zero technical support and some of the most obscure errors I've ever seen.


To the original poster...

If it's just a Django app, I'd recommend hosting on Webfaction - their support is excellent and the prices are reasonable. It's not quite as point-and-click as some of the cloud providers, but it's closer to the normal way of running a server and therefore easier to debug. Heroku is another option, which is sort of a half-way house between Webfaction and AWS/Google App Engine/etc.


It's not clear what kind of server you need but you can probably do it all with Django, and using websockets (called 'channels' in Django) if HTTP won't suffice and you need faster binary connections.


Worry about scaling later. The chance of you becoming so popular overnight that you absolutely have to be on a load-balancing auto-scaling system is minimal.

#5307603 Would You Like Fires With That? (Business Logic)

Posted by on 24 August 2016 - 07:16 AM

It's a gamble. You lose some money on not having DLC, but you gain some money from people who would have otherwise held back because they were worried about DLC. So if you already know there will be no DLC, it's probably positive. But you're unlikely to gain more revenue by saying "no DLC" than actually selling some DLC.

#5307383 My College Dilemma

Posted by on 23 August 2016 - 07:01 AM

"fresh out of college you probably don't have a AAA game under your belt" - you don't need a AAA game. There are many other levels of game which yield useful experience. As you say, side projects are useful, but so are small indie releases, or mobile games, or internships/work-experience placements/etc.


"most Designers start in QA." I don't think that's true, except at big places like Ubisoft perhaps. For a start, not every company even has a QA department (sometimes because they outsource it to publishers, or simply because they are too small).


Everyone will give you different advice about getting into game design because the role of a designer differs from company to company. (I've even worked at places that didn't employ designers at all.) What I would recommend is this: if you're passionate enough about game design to be willing to relocate to get a job, do whatever will get you the broadest and most impressive portfolio in the next few years. That is probably a game design degree, providing you can find a university with a high quality course (and all the caveats other people have mentioned apply there). A Computer Science degree will help you get a job as a programmer, but won't necessarily let you transition to designer later. But it will pay better, and give you options outside of the games industry.

#5307355 Is it inefficient to use Unity to turn 32kb of Javascript into a mobile app?...

Posted by on 23 August 2016 - 04:48 AM

I'm not sure that's so true these days - the DOM is pretty well standardised at the moment. http://youmightnotneedjquery.com/

#5307222 Is it inefficient to use Unity to turn 32kb of Javascript into a mobile app?...

Posted by on 22 August 2016 - 10:32 AM

Personally I feel that dragging in a library just for a single function is a bad decision overall. It is not elegant and often causes problems with maintanance later on.
However... I must be wrong on this because I see so many Javascript developers drag in all sorts of crazy dependencies for doing extremely simple tasks. I am sure you have already seen the word "JQuery" pop up many many times on help forums.

Hey, JQuery is usually a good choice, because it provides you with a lot of functionality that you're probably going to use later. It's a bit like seeing someone struggle with text manipulation using char*, and telling them to use std::string instead. They may not really need everything that std::string provides, but you may as well have it.


But Javascript developers do far worse things than that - you may have heard of the left-pad incident where someone removed a trivial library from the NPM package manager, one that adds a few spaces to the left side of a string to make it up to a certain length, and tons of important software was immediately broken because they depended on it indirectly.


However, this is not really an argument against using 3rd party libraries - it's an argument against:

  • deploying software by relying on every user having access to collaborative package managers
  • installing packages by name and expecting them to never be revoked, replaced, or moved
  • hiding a dependency's dependencies from the user
  • making your standard libraries so anaemic that you can't do much without pulling in 3rd party code or reinventing your own wheels
  • etc.

If a library does what you want, has already been tried and tested, and doesn't harm you or your users in some way (e.g. increased download times, difficulty of maintenance, security risks), then it's almost always more sensible to use it than to roll your own alternative.