Jump to content

  • Log In with Google      Sign In   
  • Create Account


Old Code

Posted by , 30 July 2016 - * * * * - · 1,914 views

Last week I rediscovered a 12 year old game project I was working on. I decided to look at it and write a blog about it.


Here's a summary:


Part 1: Archaeology - The rediscovery and an overview of the repository structure
Part 2: First look at the code - I open the door and peek inside, outlining the high level structure or concepts - it's a look back in time to GameDev.net in 2004 (Enginuity era)
Part 3: PhysFS & Entities - I remove the dependency on PhysicsFS to get the game starting. I explore the Entity structure and find dragons.
Part 4: Compile Times - I start cleaning up the code and apply some header discipline to drastically improve compilation times
Part 5 - It lives! - I get the game running and stare into face of a resurrected monster. I find that a lot of the systems and mechanics are no longer present or functional - as a result, the 'game' isn't a game at all anymore and won't be without significant work.
Part 6 - Cleanup - I start the cleanup of dead systems, removing a bunch of obviously unused junk. I extricate the smart pointer system in favour of a cleaner more deterministic ownership model.
Part 7 - You didn't need it - I start to remove everything else that has no material benefit to the current feature set in the game. The goal is to keep only the code that is useful. A *lot* of code is removed.
Part 8 - Journey's End - I remove the final thorn in the code, the entity system, keeping only what is needed. I realise that there's no more to remove.


It's been a fun journey and the current result is arguably the code that should have been written 12 years ago. A great lesson in not over-engineering things for the sake of it and evolving as you go. It's also a view into the past and a realisation that you can learn and change a lot in 12 years.


I'm pondering what to do next. The codebase is still "old", but improving it will result in writing a lot of new code. I may modernise it a bit and then keep going. Not decided yet. I hope you enjoy reading.

Post Mortem: Personal Labor Day "Game Jam" - Day 2

Posted by , 11 September 2015 - - - - - - · 358 views

Day 2

I started day 2 pleased with that I had a playable game at the end of the first day, but it wasn't finished yet. Cracking on, I set myself some goals for the day. The first three were continuations from the first day.

☑ Add "game over" detection (no more moves) & message

☑ Add game reset and "new game" logic

☑ Add scoring + display of scoring on screen

☑ Add high score and score load/save

☑ Add win condition (2048 tile) & message display

☒ Clean up graphics

☒ Improve user interface

☐ Add sounds

☐ Create Android build (stretch)

Game Over

Implementing this was pretty simple. For the game to be lost, you have to satisfy two conditions.
  • There are no empty tiles on the board
  • You cannot squash any tiles in any direction
I added this by keeping track of the number of occupied tiles on the board. Whenever one was squashed, it went down and one spawned, it went up. Simple. To check for "no tiles empty", it was a case of providing a property which took the total tile count and subtracted the occupied count. I added some unit tests around this to make sure the counts were ok. I'm glad I did, as I refactored something later and this broke in a subtle way..

The second check for a game over condition is the CanISquash? condition. I implemented this by duplicating the squash routines and made them return true if they would squash. This was a bit of a hack, as I should really have refactored the squashing to take a flag so that I didn't have the copy paste everywhere. I don't like code duplication, and having my unit tests for squashing should let me refactor this pretty quickly.

With the game over recognised, I needed to add the concept of "game state". I added this to my player component, but I could easily have added it to a new component. I had two states at this point.
public enum TileGameState
Adding the game over message was also simple to begin with. I created a GuiTextLabel class and a GuiRenderingService which took this label and printed the contents to the screen. I implemented a GuiService (the name sucks) which kept track of the game state and used it to modify the visibility of the text label.
 _gameOverLabel.Visible = (_player.State == TileGameState.GameOver);
This would be the start of my "battle" with the UI that took up a bunch of unexpected time.

New Game

I added a new state to my game state machine, "NewGame" which leaves us with:
public enum TileGameState
I hooked up the Spacebar key to be recognised during the game state and reset us to the new game state.

So the state transition of my game looks like.
New Game --> Playing --> Game Over --> New Game
Starting a new game currently involves several things:
  • Reset the game board to the Empty tile
  • Spawn the two initial tiles
  • Reset the game state to "playing"
In my NumberGameService update loop, I react to this:
public void Update(GameTime gameTime)
	if (_player == null)

	var player = _player.GetComponent<NumberGamePlayerComponent>();

	if (player.State == TileGameState.GameOver)
	if (player.State == TileGameState.NewGame)
		player.State = TileGameState.Playing;

	// main game update...

Scoring is really simple in this game. Whenever you squash two tiles, you score the sum of them - so squashing 4 and 4 together will yield a score of 8 and so on.

To add scoring, I went back to my player component and added "score", created a GuiLabel for the score and added it to my GuiComponent. To update this label I created a ValueFactory property, which was really just a lambda to get the value for the label. I hooked this up to some code which returned the player's score.

To actually increase the score, it meant going back into my squash code and modifying the score. I refactored out my squash code into a single method which did the move and squash for two tiles identified as being squashed.
private void SquashMoveTile(int x, int y, int nx, int ny)
	var t = GetBoardTile(x, y);
	var nextTile = GetBoardTile(nx, ny);
	if (t != nextTile)
		throw new Exception();
	SetBoardTile(x, y, t+1);
	SetBoardTile(nx, ny, TileType.Empty);
	_score += ScoreForTile(t + 1);
Because you can squash multiple tiles in a single move, I chose to let the squash code modify the player's score directly. I wish I'd not done this as it's pretty ugly - I should have made this method return the score and then kept track of it locally. It's pretty easy to clean up, especially with my tests in place Posted Image

At this point, my game looked like this (yes, it still looks bad).

Attached Image

High Score & Profile Saving

Adding high score was pretty easy. When the game transitions to Game Over, check the current score against a high score and update it if needed. When a high score is detected, then set a flag so that a message can be displayed on the UI. I also needed a new "high score" label too. This lead to code which looked a bit like this:
_playerScoreLabel.Text = string.Format("{0}", _player.Score);
_playerHighScoreLabel.Text = string.Format("{0}", _player.HighScore);
_gameOverPanel.Visible = (_player.State == TileGameState.GameOver);
if (_gameOverPanel.Visible)
	_gameOverLabel.Text = "GAME OVER";
	_gameOverNewHighScoreLabel.Visible = _player.NewHighScore;

As I was now grouping up labels, I added the concept of a "panel" to my GUI. It basically acts as a container for other elements and becomes their origin point; so.. gameOverPanel contained two elements - gameOverLabel and gameOverNewHighScoreLabel. Both had positions relative to the main gameOverPanel, so I can move this panel and also move the child elements. This is the beginning of an actual UI system, and as I said earlier - the start of some head scratching.

Now I had a high score, I needed to save it and load when the game starts. To do this, I used the XNA StorageDevice subsystem and serialized a "SaveGame" class to json using JSON.Net. Sure, people can cheat and change this - but who cares, right?


To win in 2048, you must create the 2048 tile. I implemented this by keeping track of the highest tile created when squashing. As soon as that tile hits 2048, the game can transition to Game Over, setting a "won" flag. It's a bit of a hack, but I used this flag to change the game over message to "You Won!" if this flag is set - and that's it.

With this in place, I had a fully playable game. You can win, lose and try and beat your own high score. I actually found myself playing it a lot during dev and it felt pretty close to the 2048 experience.


Let's face it - I'm no artist. Making things look nice is hard for me but it's the thing that makes the biggest difference to how people see your work. The first thing to do was to update the tile set. I ended up with this...

Attached Image

Not perfect, but it does the job.

The next step was to add some UI colors and a "skin".

Putting this together, I ended up with:

Attached Image

Looks a bit better than the "Mother of God!" original, but still not ideal. But it's the best I could do in the time.

I cheated on the skin; instead of 9-slicing the UI skin and using each element in the correct place on the panel (corners, sides, middle) I cheated and used a panel texture. This creates horrible scaling artefacts for larger panels, as you can see here:

Attached Image

That's probably simple to fix; but I've not done it yet.

User Interface

To this point, the UI is built from 5 primitive classes.
  • GuiElement - the base element
  • GuiElementContainer - a collection of elements
  • GuiCanvas - the main UI parent (it is a container)
  • GuiPanel - the sub panels you see, with textures and color
  • GuiTextLabel - the text labels
Using these classes, I could compose the UI you saw earlier.

Here I was using absolute positioning for pretty much everything and realised that if I wanted to ship on Android, or even different Windows screen sizes, I'd need to make my UI scale.

So I sat about redesigning it. I had a look around at the various ways other people have done it. Notably:For this iteration, I settled with something comparable with CEGUI and Urho3D. That is, my co-ordinate system has a dimension which has both Scale and Offset. These dimensions are used for positions and sizes, allowing me to create elements that are 50% of their container size and positioned in the center. Additionally, I can add an offset to this. Part of doing this was to add things like text centering for my labels, which work - but is crude, and doesn't work with multiline at the moment.

Implementing all of this took much longer than I expected and basically chewed up all of my remaining polish time and it's still far from ideal. I didn't get to add sounds, for example, and I have no player feedback systems in place except the basic movement.

Where I am with this right now is that it's functional, but still has a bunch of shortcomings. For starters, I have to code my UI - I don't yet have layout xml files. Also, I don't have basic things such as min/max sizing or margins. I'd like to aspire to implementing something like the new Unity 5 system, which basically allows me to anchor corners to a part of their parent, allowing nice scaling. My UI still doesn't scale very nicely and it needs work before I can even think about letting it loose on a non-fixed screen size that I control.


Even a simple game, such as 2048, can still take a bunch of work when creating it from scratch. The biggest thing I got from this is that I feel like I balanced my goals of the jam with the end result fairly well. I wanted a full functional game and some tech that I could potentially reuse next time. I feel I achieved both of these things. Sure, I could have created this game in 4-6 hours in Unity or Unreal and I wouldn't have had any of the UI issues or anything else that caused me problems, but that wasn't the goal here.

Overall, I was relatively happy with the architecture of my game engine. It was really quick to iterate on and felt comfortable to work with. Sure, the Entity-Component and Service paradigm may have been overkill for 2048, but it ended up working well and leaves me with something to build on next time. I think the only real area that was problematic was that of the UI, but that's something I'll look at again next time around.

I did want to add a game over "screen", as well as other menu screens (see last N game history, etc), and it became apparent that because my systems weren't designed around screens or priorities, I had no clean way of filtering inputs, or pausing the gamestate. This is something I will seek to address for the next game on this codebase.

I was happy with how I structured my process. My goals nicely lead into each other and provided something tangible when they were achieved. Having something on screen with the "Mother of God!" artwork was crude, but it let me accelerate to "playable" very quickly - and I was happy with that. I am also glad I put some unit tests around my code logic; it let me iterate over the rules quickly and with safety, providing me early feedback when I break something.

One small snag was that I was brand new to Pyxel Edit so was learning how it worked as I went on. I have a few niggles with it, so I plan to keep using it after this to get the hang of it.

All in all, I had a bunch of fun doing this. The game may be shitty but it's created from scratch and is as playable as the game it clones. For the first time in years I got into the "flow" with my own projects and went with it for two days.

Take Homes

Here's something that I'm taking with me and could be used as broader advice to people wanting to do something similar (or for total beginners, in fact).

☑ Know your goals up front

☑ Make sure your goal is realistic for your ability/knowledge

☑ Make sure your goal is realistic for your time

☑ Pick your tools wisely; use things you feel productive in or can learn quickly

☑ Iterate quickly to playable, with horrid art all the way

☑ Add tests around critical systems, they will help you later

☑ Don't underestimate things such as user interfaces; they can become a sinkhole

☑ Know when something is "good enough" and move on to the next thing, come back to things when you need to

☑ Have fun!

As a side note, the game 2048 is a great project for a beginner to game development to take on. It covers most things you would need to do in a game and provides a couple of interesting challenges in how you might want to execute (and optimize) the moves.

Next Steps

As this turned out to be pretty successful, I'm planning on looking at making another game. Perhaps a Space Invaders or Galaga clone using bits I can salvage from the codebase (which should be fairly high).

I may also write a follow-up post about the technical design of this codebase.

Post Mortem: Personal Labor Day "Game Jam" - Day 1

Posted by , 10 September 2015 - - - - - - · 335 views


It's been a long time since I've done anything directly game related in my spare time. I work 40+ hours a week on video games and by the time I get home, I often can't summon up the motivation to do anything dev related at home.

Sure, I've still got ideas and desires to work on projects I've carried in my head (or as failed prototypes) for literally years, but when I sit down to work on them at home the mojo never seems to be there.

I was interested in participating in the recent Ludum Dare 33 game jam, but never go round to it - and I regret not doing it.

So, over labor day I decided to sit down and make a game; the simplest game I could put together in 48 hours with only me and no support.

Because I enjoy a little bit of the tech work (engines, frameworks, etc) I chose to start from scratch in MonoGame instead of going the traditional Unity/Unreal 4 route. I also wanted to make something that resembled a decent architecture and not just a smashed together hack - primarily because I intend to do this again and wanted a base to work from.

The Game

The game I picked was 2048 - the crappy mobile game from Kerchapp (which is a clone of 1024, which is a clone of Threes). If you've never played it, it's a game about matching pairs of numbers together to make their double, and repeating until you make a 2048 tile.

The reasons for picking this game:

☑ I play this game

☑ Easy game mechanics

☑ Minimal graphics requirements

☑ Minimal audio requirements

☑ Achievable in a weekend

☑ I stand a good chance of finishing it

The Tools

To do make this game, I used the following tools:I didn't get around to implementing audio, so didn't use any tool for that.

Day 1 Goals

I didn't make a concrete plan for putting this together, but I did set mini "goals" for what I wanted to see.

Here's my list of day 1 goals and which were achieved.

☑ Lay down the foundations; resource system, entity system, component system

☑ Throw together some crude graphics and get a game board on screen

☑ Implement the input system & basic movement rules for tiles

☑ Implement the "squashing" of two adjacent tiles when you move them together

☐ Add "game over" detection (no more moves) & game reset

☐ Add scoring + display of scoring on screen

Implementing the basics of the framework were pretty simple. I put together an architecture I've played with in the past - the Entity-Component & Services architecture. The idea of this is that we have GameObjects (entities), which have Components attached. Unlike systems such as Unity, my components don't contain logic (or if they do, it's calculations). Instead, logic is collected in GameServices, which have a Start/Stop/Update method.

To tie all this together, I have a simple Event system, which passes messages around between anyone who subscribes to a given type. I implemented a class called a GameObjectComponentView, which subscribes to the GameObjectComponentAdded/Removed messages (and GameObjectDestroyed) and filters them based on specified component types. This object then maintains a list of objects it knows about, allowing you to enumerate over them knowing that each of them have the components you need. This system formed the basis of most of my systems.

To draw the main game grid, I decided to use a TileMap. I created a crude TileMap2DComponent which held the state of the board (4x4) and held some texture atlas information about how to draw each tile. All of the game board tiles were held in a single texture, with each tile being 64x64 (too small). My texture map was 2 layers, one which had the actual pieces and the other which had a "frame" for each piece. I could have drawn the pieces with the frame over them, but I was lazy Posted Image

Tying this all together, I created my game service called NumberGameService, which used the TileMap2DComponent and my new NumberGamePlayerComponent.

Input System

Now I had something on screen, it was time to start thinking about inputs. I followed the standard approach of creating a simple "state map" and "action map". Essentially I had a json file which describes the key actions (press/unpress) and states (pressed/not pressed) and how to map them to game actions.

A snippet of this file is here:
    "ActionKeys": [
        { "Key": "Escape", "Action": "action.quit" },
        { "Key": "Space", "Action": "action.newgame" },
    "StateKeys": [
        { "Key": "Up", "Action": "action.move.up" },
        { "Key": "Down", "Action": "action.move.down" },
        { "Key": "Left", "Action": "action.move.left" },
        { "Key": "Right", "Action": "action.move.right" },
My input service would then monitor these keys and raise events with the corresponding action in them. For the day 1 implementation, I raised an event for each one - but this was changed in day 2.

Now I could raise an event based on input, it was a case of hooking my systems up to do this. The Event Manager I wrote could handle this with ease; it was a case of making my NumberGameService subscribe to specific events and pass it a lambda/delegate to handle it, eg:
_context.EventService.Subscribe<InputActionEvent>(this, (evts, e) =>
    // do stuff
This proved to be pretty flexible for everything I needed - but I did optimize it a bit later.

Game Logic

With inputs ready to go, I finally started to implement the game logic. Initially, I started this in my NumberGameService but it rapidly became painful to verify and iterate on. To address this I created a NumberGameBoardLogic class, which had the methods MoveLeft()/MoveRight()/MoveUp()/MoveDown() and would modify the NumberGamePlayerComponent and the TileMap2DComponent. With this class, I set about writing some Unit Tests, whereby I could set the state of the board to some known state and then simulate a move, verifying it did what it was supposed to.

An example would be:
public void MoveRight_MovesTileWhenSingleTileCanMoveRight()
	var uut = CreateUut((a, b) => 0);

	uut.SetBoardTile(2, 0, TileType.Number1);
	var result = uut.Move(PlayerMovementState.Right);
	uut.GetBoardTile(2, 0).Should().Be(TileType.Empty);
	uut.GetBoardTile(3, 0).Should().Be(TileType.Number1);
It's not a great example of a test, but I wrote several tests to cover all the various scenarios I could see. This proved incredibly useful to picking up places I broke the code when I changed things - especially when I came to add scoring. If anything, I wish I'd created far more tests than the ones I did.

Squashing Tiles

With the movement set up, I was ready to implement "squashing" and therefore scoring. To do this, I went back to my unit tests and started implementing the tests to show the results, then I went back and fixed the code to pass the tests. This proved to be useful, as I broke a few things that would have taken me a while to find!

The move and squash logic is brute force and isn't optimised; but for this project it was good enough!

It goes something like this:
bool MoveDown()
	var score = _score;
	var moves = DoMoveDown();
	if (_score - score <= 0)
		return moves > 0;
	return true;
In case you don't know how 2048 works; a move basically pushes all the tile to extreme of the direction you tell it. So if you had a row like such:
|2| | |2|
And moved right, the result would be:
| | |2|2|
But then because these tiles moved together and have the same number, they get squashed...
| | | |4|
There is a second move if a squash has occurred because you can end up in situations like this:
Should become:
| | |4|4|
And not:
| |4| |4|
Or even:
| | | |8|
After each move; the game spawns a new low tile in an empty spot and you keep going.

With all this implemented, I had something playable! And I ran out of time - it was getting late and I needed a screenbreak. I felt pleased that I had a playable thing at the end of the day - and it was as fun as the 2048 game.

It's worth noting that it looked like shit. Here's the programmer art screenshot of how the game looked at the end of day 1...

Attached Image

To quote @EGVRoom when he saw this image... "Mother of God!". And it's true, it was bad.

With Day 1 wrapped up, I laid down a few goals for Day 2 - implement the Game Over/Win conditions (to make it a "game") and to polish what I had.

But I'll leave that for my next entry.

Chamois - Fluent Assertion Syntax for C++

Posted by , 13 September 2014 - - - - - - · 2,368 views
c++, unit, test, bdd, tdd, fluent
One library that I love in .NET is Fluent Assertions. It takes assertion syntax in Unit Tests and wraps it up in a natural language syntax. Doing so, we get a nice looking structure for our test that reads like a requirement, rather than a bunch of code.

For example:

assert(my_variable == true);

Sure, it's nice and tearse for a programmer, but what if it were more fluent?


Or a scenario where we want to check a range:

assert(!(my_variable >= min_value && my_variable <= max_value);

Oh crap, now we're getting more complex! But with a fluent syntax, it makes it more readable.

Assert::That(my_variable).Should().BeInRange(min_value, max_value);

Inspired by the brilliant .NET Fluent Assertions, I created Chamois, a header-only Fluent Assertion library for C++ Unit Testing.

The primary ways of expressing a test is as follows:


I currently support the following types:
  • Integral numerics (short, int, long, float, double)
  • String (via std::string/std::wstring, including const char* / const wchar_t*)
  • Arrays (simple arrays)
  • Pointers (naked pointers only)
  • Any object by reference that supports the equality operator
Support for more standard library containers, smart pointers, etc is planned.

Currently, I only support the Microsoft's test framework assertions but I'll be adding more soon - including c-style assert for simple tests.

On C++ Naming Conventions

Posted by , 15 July 2014 - - - - - - · 4,068 views

I threw myself back into the deep end of C++ again a few months ago, having spent the last couple of years with an emphasis on C# and .NET.

One thing that I'm thinking of is the old subject of coding style. Every software house tends to have a house style, so at work you just adopt that - but at home, on personal projects I find myself drifting around stylistically, at least with naming conventions.

There's a few main styles that I tend to consider ok for me:

.NET Style
Microsoft have a standard set of style guides for .NET / C# programs which for the most part people adopt (although not always, even within Microsoft public code).

The .NET style is simple:
  • PascalCase for class names
  • PascalCase for public methods (all methods, in fact)
  • Interface classes start with an "I"
  • Public properties/variables are PascalCase
  • camelCase for variable names
  • Namespaces are PascalCase
  • Assembly Names are PascalCase
Code would look something like this:
namespace MyProject
    public interface IBar { }

    public class Foo : IBar
        public void DoSomething(int parameterName);
        public int SomeProperty { get; set; }
It's worth noting that the get/set property functions end up becoming something like "SomeProperty_get"/"SomeProperty_set" under the hood.

Java Style

Java programs also have a common style.
  • PascalCase for class names
  • Interface classes start with an "I" (but not always)
  • camellCase for public methods
  • Public properties/variables are camelCase, prefixed with getXX() or setXX() (as Java doesn't have properties)
  • camelCase for variable names
  • Namespaces are lowercase
  • Package Names are lowercase
In Java, the above example looks something like:
package myproject;

public interface IBar { }

public class Foo implements IBar
    public void doSomething(int parameterName);
    public int getSomeProperty();
    public void setSomeProperty(int value);
C Style / C++ Standard Library Style

C++ seems to have a tonne of styles. One of the first ones you'll come across is that of the standard library, which appears to have adopted the C style convention, largely due to remaining consistent with the old code from yesteryear.

Here, we have:
  • lowercase_delimited for class names
  • Interface (abstract) classes aren't special
  • lowercase_delimited for public methods
  • Public properties/variables are lowercase_delimited, there doesn't seem to be a get/set style (?)
  • lowercase_delimitedfor variable names
  • Namespaces are lowercase
Back to our example:
namespace myproject

    class bar 
       virtual ~bar() { }

    class foo : public bar
        void do_something(int parameter_name);
        int get_some_property();
        void set_some_property(int value);
Looking through some other sources and the Google C++ guide seem to lean to a blend of the Java Style with C++/Standard library style.

  • PascalCase for class names
  • Interface classes start with an "I" (but not always)
  • PamellCase for public methods
  • Public properties/variables are lowercase_delimited, prefixed with my_property() or set_my_property()
  • camelCase for variable names
  • Namespaces are lowercase_delimited
This leads to:
namespace my_project
    class Bar 
       virtual ~Bar() { }

    class Foo : public Bar
        void DoSomething(int parameterName);
        int some_property();
        void set_some_property(int value);
With all this, the Google version seems to make a lot of sense.

What's your style and how did you pick it?

First version of FlatBuffers in .NET

Posted by , 05 July 2014 - - - - - - · 1,143 views

I've committed by first alpha version of the FlatBuffers port to .NET to GitHub.

Since my last post, I decided to port the Java version to .NET as straight as I could, which means the use between Java and C# should be similar.

I ported the JavaTest buffer creation code; which looks as follows:
            var fbb = new FlatBufferBuilder(1);

            // We set up the same values as monsterdata.json:
            var str = fbb.CreateString("MyMonster");

            Monster.StartInventoryVector(fbb, 5);
            for (int i = 4; i >= 0; i--)
            var inv = fbb.EndVector();

            Monster.AddHp(fbb, (short)20);
            var mon2 = Monster.EndMonster(fbb);

            Monster.StartTest4Vector(fbb, 2);
            MyTest.Test.CreateTest(fbb, (short)10, (byte)20);
            MyTest.Test.CreateTest(fbb, (short)30, (byte)40);
            var test4 = fbb.EndVector();

            Monster.AddPos(fbb, Vec3.CreateVec3(fbb, 1.0f, 2.0f, 3.0f, 3.0,
                                                     (byte)4, (short)5, (byte)6));
            Monster.AddHp(fbb, (short)80);
            Monster.AddName(fbb, str);
            Monster.AddInventory(fbb, inv);
            Monster.AddTestType(fbb, (byte)1);
            Monster.AddTest(fbb, mon2);
            Monster.AddTest4(fbb, test4);
            var mon = Monster.EndMonster(fbb);

My test code will read a file output from C++ and assert against it being valid (readable). Then create the same buffer in .NET and run the same tests against it; exactly like the JavaTest does. My .NET version passes both sides, so it means we should be compliant to the FlatBuffer format and can achieve interop between C++ and Java.

By being in C#, this should be usable from Unity - either as code, or binary. I haven't verified this yet though.

Now that I've got a working version, I can start to address my other concerns - that it doesn't "look" nice to .NET developers ;)

What this really means is that I'll be building a reflection based serialization layer on top of this, so we can do code-first and/or transparent POCO serialization from the FlatBuffer format.


After thinking about this, there's a few things I want to look at improving. The first version was a straight port, and this works fine but sticks out in C# like a sore thumb.
  • Modify the C# code generator to create a property for get access (eg: not monster.Hp(); but monster.Hp; )
  • Change the generated C# objects to XXXReader and XXXWriter; this lets me have cleaner split of read/write concerns
  • Refactor the FlatBufferBuilder into a FlatBufferReader and a FlatBufferWriter - again, to separate the concerns a bit
  • Look into a nicer way of handling arrays/vectors
  • Build a "typemap", effectively a definition of a FlatBuffer type which can be used to serialize or code gen POCOs
  • Add the Serialization support for POCOs (mentioned above)
  • Add a MediaTypeFormatter for ASP.NET / etc to allow simple model binding with FlatBuffer format

FlatBuffers in .NET

Posted by , 24 June 2014 - - - - - - · 4,048 views
.net, c#, protobuf, flatbuffers and 1 more...
I've been tinkering with Google's new FlatBuffers protocol and have an experimental port of it running in .NET (C#).

FlatBuffers is interesting and there's a couple of other alternatives in the form of ProtoBuf, Cap'n Proto, Simple Binary Encoding (SBE) and, to a lesser extent, JSON, XMLand YAML. These protocols all share the same thing in common, to serialize your data into a format that's portable between applications - be it as an on disk format (save game, level, etc) or as a wire object for communicating with network services.

JSON is really the go-to choice now for people who use web services a lot and works well as a configuration file format too. However I've recently been noticing a trend towards binary encoded data, especially in the arena of game (client/server, especially); the reasoning behind this is often due to performance of both encode/decode and that the text-based formats take up more network bandwidth.

One issue with binary protocols has always been that it can be extremely easy to change the schema and render the existing data invalid.

Take the simple object:
class Vector3
    float X;
    float Y;
    float Z;

class Monster
    int Health;
    Vector3 Position;
We would probably serialize out with something like this:
void SerializeMonster(std::ostream &stream, const Monster &monster)
   stream << monster.Health;
   SerializeVector3(stream, monster.Position);

void SerializeVector3(std::ostream &stream, const Vector3 &vector)
    stream << vector.X << vector.Y << vector.Z; // Quantized floats omitted for simplicity

And deserialization would be something like this:
Monster* DeserializeMonster(std::istream &stream)
   auto monster = new Monster();
   stream >> monster.Health; 
   monster.Position = DeserializeVector3(stream);
   return monster;

Vector3* DeserializeVector3(std::istream &stream)
   auto vector = new Vector3();
   stream >> vector.X;
   stream >> vector.Y;
   stream >> vector.Z;
   return vector;
If I wanted to add additional fields to the Monster class (Mana, Color, Weapons carried, etc), I would have to be very careful to keep my functions in sync, and moreover if I change the serialization order of anything all my stored data is useless.

This problem gets even bigger if you start talking outside of a single application, such as sending data from your client to a network server that might have been written in .NET, Java or any other language.

A few years ago Google released V2 of their Protobuf format which aims to bring a bit of sanity to this problem. They defined their serialization objects in a simple IDL (interface definition language). The IDL used by Protobuf has a formal native type format and then your data is defined as "messages". Protobuf has a "compiler" which then generates your C++ (and other languages) serialization code from the IDL.
message Vector3 {
  required float x = 1;
  required float y = 2;
  required float x = 3;

message Monster {
  required int32 health = 1;
  required Vector3 position = 2;
A key thing to note here is the numbering used; this defines the serialization order of the field. Protobuf mandates that if you want to maintain backwards compatibility, you must not change the ordering of existing items, new items are appended with a higher sequence and they are added as "optional" or with a default value. Anyone interested should look here for more details on this.

Protobuf does some clever encoding of the data to minimize the size; things such as packing numeric types and omitting null values.

Projects like FlatBuffers (and the others mentioned above) cite that the encoding/decoding step of Protobuf is expensive - both in terms of processing power and memory allocation. This can be especially true if done frequently - such as if your network server is communicating in protobuf format. In some of the online services I've worked on, serialization to/from the wire format to an internal format has been an area to focus optimization on.

FlatBuffers, Cap'n proto and SBE take the stance that the data belonging to the object is laid out the same in memory as it is on disk/during transport, thus bypassing the encoding and object allocation steps entirely. This strategy becomes an enabler of allowing simple mmap() use or streaming of this data, at the expense of the data size. I'm not going to go in the pro/cons here, as the Cap'n Proto FAQ does a better job. However, all of these in-memory binary data formats all acknowledge that having a "schema" which can be modified and retain backwards compatibility is a good thing. Basically, they want the benefits of Protobuf, with less overhead.

So back to the topic of this post; how does .NET fit into this? Given that engines such as Unity provide C# support and XNA / MonoGame / FNA are all managed frameworks capable of targeting multiple platforms, FlatBuffers in .NET has a couple of obvious use cases:
  • Data exchange between native code and C# editors/tools
  • Game data files with a smaller footprint than JSON/XML/YAML
  • Wire data exchange between Flatbuffer enabled client/servers (to a lesser extent, due to bandwidth concerns, but viable in the same datacentre)
  • Interprocess communication between .NET and native services (same box scenarios)
I started porting the Flatbuffers protocol to .NET and hit a slight dilemma; should we treat Flatbuffers as a serialization format, or should it also be the "memory" format too?

In Marc Gravell's Protobuf-net project he explicitly took the stance that "rather than being "a protocol buffers implementation that happens to be on .NET", it is a ".NET serializer that happens to use protocol buffers" - the emphasis is on being familiar to .NET users (for example, working on mutable, code-first classes if you want)."

Right now, I'm facing a similar question. What is more important for .NET users? Is it the ability to code first and serialize, or to be able to access Flatbuffer data without serializing?

Here's an example of the code-first approach:

public class Vector3
	public float X { get; set; }
    public float Y { get; set; }
    public float Z { get; set; }

public class Monster
    public int Health { get; set; }
    public Vector3 Position { get; set; }

var someBufferStream = ...;

var monster = FlatBuffers.Serializer.Deserialize<Monster>(someBufferStream);
We declared our various classes and used an attribute to annotate them. Then we explicitly take data from the stream to create a new Monster (and Vector3) object. From here, we can discard the buffer as we have our .NET objects to work with.

Serialization would be the opposite: create your objects in .NET and call a Serialize() method to fill a Flatbuffer.

The benefits of this are that is it is very .NET centric; we are used to code-first approaches and serializing to/from our entity classes. Conversely, we now have to either manually synch our IDL files, or write a tool to generate them. We also begin allocating many objects - especially if the buffer is large and has a lot of depth.

An alternative approach is the accessor pattern which the FlatBuffers Java port takes. Here, we effectively pass a reference to the data in Flatbuffer format and use a accessor methods to read what we need. The accessor classes would be generated by the IDL parser tool.
public class MonsterAccessor
    public MonsterAccessor(FlatBufferStream flatBuffer, int objectOffset)
	 { .. }
	public int Health { get { flatBuffer.ReadInt(... objectOffset + offset of health ...); } }
	public Vector3Accessor Position { get { return new Vector3Accessor(flatBuffer, flatBuffer.ReadOffset(... objectOffset +  offset of position ...)); } }

public class Vector3Accessor
    public Vector3Accessor(FlatBufferStream flatBuffer, int objectOffset)
	 { .. }
	public float X { get { flatBuffer.ReadInt(... objectOffset +  offset of x ...); } }
	public float Y { get { flatBuffer.ReadOffset(... objectOffset + offset of y ...); } }
	// etc

var someBufferStream = ...;
var offset = ...;

var monsterAccessor = new MonsterAccessor(someBufferStream, offset);
var monsterHealth = monsterAccessor.Health;
var vector3Accessor = monsterAccessor.Position;

// etc

In addition to the Reads, we would also supply "Write" methods to populate the buffer structures.

The benefit of this approach is that the buffer access becomes lightweight; we are no longer deserializing whole objects if we don't need to - we're not allocating memory (except for things like strings). We also get direct sync with the IDL version.
The downside of this is that working like this feels quite alien in the .NET world; the code to handle buffer access is very manual and in the case of the position access, very clunky. It's likely that we will end up creating lots of accessors - unless we rely on everything being static. Which also becomes pretty... nasty.

This latter approach is more akin to a "port" of Flatbuffers, rather than treating it like a serialization type.

So - with these approaches in mind; what would YOUR favoured access method for FlatBuffers be in .NET? Would you focus primarily on FlatBuffers as a serialization format, much like Marc Gravell did with protobuf-net? Or would you look at a true port? Or maybe there's some hybrid approach?

What would your use cases be?

Porting Accidental Noise Library to .NET

Posted by , 14 April 2014 - - - - - - · 1,475 views

I've been inspired by the images that JTippetts' Accidental Noise Library can generate. However, these days I do very little C++ work so I felt like tinkering with a little project that would help me tip my toe back in this water and also have a bit of fun.

As a result, I decided to have a crack at porting Accidental to .NET. I'm aware that James Petruzzi already done a partial port and that Mike Tucker has integrated this partially into Unity3D, but this wasn't the exercise for me.

I wanted to a) have a little play with the noise b) understand the library and routines a little more and c) tinker with C++ again in a gentle porting project. If I have something cool or useful at the end of it, then it's a bonus.

Accidental is quite a nice system. It's effectively a series of modules that you configure and then chain together to get some pretty pictures out of the other end. Most modules typically have a "Source" and return results via a "Get" method; as such, the results of one module typically feed into another and so on until you get something useful out the other end.

The original Accidental library supports 2D (x,y), 3D (x,y,z), 4D (x,y,z,w) and 6D (x,y,z,w,u,v) modules/generators all off the same base. In my current implementation, I'm deliberately splitting the various parts into their various dimensional constituents. I don't know yet if this is the best approach, but it's making life easier for me.

I've ported over a bunch of stuff so far; but there's plenty to do.

In the original Accidental Noise Library, JTippetts used Lua to configure his pipeline. I'm going to look at using alternative, more C# friendly solutions to this - possibly JSON Posted Image

Right now, everything's just in code so seeing the results of a tweak can be a little slow.

Here's some pictures:

Attached Image

Fractal: Billow (Default), autocorrected to -1,1 range and scaled to 6,6.

Attached Image

Cellular: Scaled to 6,6

Attached Image

Fractal: fBM, frequency 3

Attached Image

Fractal: Rigid Multi, octaves: 3, scaled to 6,6

I'll probably push the source up to GitHub at some point soon.

Recent Comments