A Hobbyist Game Engine Post Mortem	GameDev.net

A Hobbyist Game Engine Post Mortem
by Paul Coppens

Context

After spending several years developing mostly 2D/3D small applications in different languages on different platforms, I decided to create ex-nihilo, an object-oriented game engine with 3D graphics. My main reasons for creating my own engine were:

I already had existing code samples (graphic display, skeleton based model, sprite, …)
When comparing with existing engines at that time, I thought they were lacking simplicity compared to the kind of programs I wanted to write and did not offer direct support for sprites, fonts and/or user-defined GUI.

I also wanted to learn how to develop a cross-platform engine with no external dependencies (except the Standard C++ Library). I was considering whether I should continue developing under Windows or switch to Linux and I didn’t want to have too much work porting the engine. I also wanted to improve my refactoring skills. This endeavour as a lone hobbyist game programmer took me a little more than 1500 man hours over two years. Here is the result of my developments.

What Went Right

1 - Focus

I wanted a cross-platform game engine to realize 2D/3D turn based strategy games or simple RPG games like Final Fantasy 7. As such the preliminary features were:

OpenGL for cross-platform graphics at least v1.2.
Object-Oriented Design
Scripting: simple to change on-the-fly combat resolution, to code spell/weapon effects and game events
Import/Export a few formats: the Milkshape 3D modeller text and binary format + my own internal format for sprites and models, TGA, BMP and light length-encoded bitmaps.
Simple physics (basic collision detection of Axis Aligned Boxes and spheres; basic navmesh handling).
Hardware vertex transform and lighting (T&L)
Basic Texturing
Skeletal Animation
Special Effects: Billboarding, Particle Systems, Simple Terrain Rendering (Hex based and square based),
Fonts and GUI support
Sound system (mod player).
Embedded logging system.
Basic Multithreading availability

Each feature was simple enough to be realized and there existed an ample amount of documentation on the Net and in books. Moreover, I got a moral boost each time I completed a feature on that list.

2 - Planning

In my day-time job, I am used to writing software/process specifications and planning schedules. This made the process more natural to do as I knew how. Moreover, I knew what time I had available to me and planned in consequence. Even though my documents were less detailed than what I do at work (hey, that’s a hobby for me), I had defined goals to reach and found out it would require one year’s time to complete the project. I knew from the beginning it was no trivial task and was prepared for it.

Here is the simple planning I did:

Design the architecture: 2 weeks
Realize the core framework (interface for creating a blank Window + initialising OpenGL + basic helper functions and logging + basic multithreading interface): 3 weeks.
Realize the basic core Renderer (frustum + renderer + math interface + Color and Light interfaces + texture manager): 6 weeks.
Create the font engine (refactor bitmap and rendered font code, include default fonts, create font file format): 4 weeks.
Create the basic Scenegraph (Scenegraph and basic volume generator): 4 weeks.
Create the Mesh Model Node (refactor existing code and file loading, process animated and static models + model manager): 5 weeks
Create the Sprite Model Node (refactor existing code and file loading, create billboarding code, process animated and static sprites + sprite manager): 3 weeks
Create the Particle engine Node (create code, create interface): 3 weeks
Create basic terrain rendering (refactor terrain generation code with hex and square cells, create file format): 2 weeks.
Create the GUI interface (refactor existing code for buttons, dialog, text, progress bar, mouse widgets, create file formats for skinning and GUI loading): 5 weeks
Create the scripting interface (refactor existing code for scripting): 2 weeks.
Create the sound system (reuse DUMB and adapt with OpenAL): 10 weeks.

Note that to me, a week is composed of 1 to 2 hours of programming per day with 7 days in a week (at work during lunchtime for one hour + one or two hours in the evening). Each of the tasks planned were undividable but could be realized in any order (except the 5 first tasks).

3 - Careful design

Before committing to development I took time to polish my engine architecture. Contrary to small scale projects, this one needed to be carefully thought. It took me about 3 weeks to define a satisfying UML diagram of the engine. It did pay in the long run as each new feature integration neither broke nor frankly stressed the design. So no mid-project code overhaul or architecture redesign was ever needed. This also helped refactoring previous code as I knew the interfaces and how it should interact with the engine. This was another proof of a sound design adapted to my needs.

4 - Previous experience

I was able to capitalize on all my previous experience. I had already written simple expert systems, a forth interpreter, 2D graphics renderers, 3D mesh viewers including skeletal animation, basic multithreaded resource loaders… all simple applications in different areas of the game engine I wanted to develop. This gave a great boost to the realization of this game engine and improved my refactoring skills. This confidence is also reflected from the beginning in the planning where I knew with great confidence it was not uncharted areas. Good for the morale to know before hand that tasks won’t be that difficult to do.

5 - Iterative development

To me, what was of utmost importance was that each new feature and optimisation added did not add bugs to the engine or degrade performance. So I wrote a few samples using the engine (and even editors) and each time I modified the code, I compiled the engine against all examples to check that they perform as they were designed to. All examples were also improved each time to include the latest added features: they doubled as engine tests. I also ensured the engine and the examples worked the same way on all three different computers I use:

an IBM ThinkCentre 8429 with no added graphics card using Windows 2000.
a home-made Pentium 4 2Ghz with an old GeForce graphics card using Windows XP Professional edition.
a laptop AMD processor with an internal GeForce 780X graphics card using Windows XP Home edition.

Plus I established the policy that bug resolution was to be done prior to any further development. I had activated the “signal all warnings” option on my compiler to be sure to catch bugs early on. Engine stability was really important. Discipline is the best friend of the developer. This iterative development led to a very robust engine and a very clean library.

6 - Engine documentation

As this was a really big scale project, I wanted to easily maintain code documentation for ease of reference in later developments using this engine. I learnt to use Doxygen and never went back. It really was simple to maintain documentation (instead of writing full descriptive documents under Word which were a pain to maintain). This also improved the reuse of the engine since I was able to develop samples two years after the beginnning of the engine development.

7 - Keeping track

I also manually kept a log of any changes I made to the engine in a simple text file. I also manually logged any bug encountered in another text file. Bug correction had higher priority than feature development. This log helped me to keep track of what I was doing and where I stopped (invaluable for a hobbyist programmer with little development time available) but also gave a moral boost when considering everything that was done and all steps ensuring the stability of the engine.

8 - Logging for debugging

When developing a multithreaded application, the only solution to trace a code or to time a code is through logging. I had explored the use of gprof for debugging purposes. Even though it is mandatory for single thread applications, I saw two drawbacks for multithread applications:

often the debugger does not stop on the spot where the error occurred. Moreover the spot where it stops changes over some application execution.
a few times the error was not stable when the debugger was on: the program could run normally without any bugs on one execution and crash on another.

The embedded logging system with different levels of logging available proved to be invaluable. Moreover, I added a trace method to some classes for easier debugging information on the engine internals. Debugging state changes for optimisation was easier. The logging system was also available when integrating this library in a project. It eased the optimisation of the display and the tracking of state changes (or identify faults when loading in resources).

What Went Wrong

1 - Creeping features

Even though I had carefully laid out my plans, I could not help adding a few more features. I added:

To AI scripting system: basic subsumption and rule-based system.
To texturing: multitexturing and bumpmapping.
To rendering: render to texture interface.
Faster font rendering.
PNG bitmap loading.
OFF mesh format processing.
A XML file parser.
A scenegraph file format.
A curve generator using three different methods

I stopped myself before I added any more features: I then was considering mirror rendering and thinking about basic shadows. Creeping features always occurred when looking at/playing other games (a shame I tell you ? ). I wanted to add more chrome to my display (bump-mapping, render to texture), to diversify object loading for simpler game resource integration without using third party converters (PNG and OFF for example).

I stopped myself when I understood I was thinking more in terms of engine elements than direct implementation. This proved to myself that I had finally come to understand how a graphics/game engine works and how I can extend such an engine, even if it is a third party engine.

2 - Not made here syndrome

Under the guise of learning things (or not having external dependancy), I discovered I was in fact writing code and reinventing the wheel. I wrote for example a file system which let me indifferently load a resource from a package or from a physical disc. Even though it worked, it was of lower quality than code available on the Net which was better realized, cross-platform and compatible with multithreading (like PhysicsFS by Ryan C. Gordon for file system abstraction).

After discovering this, I created interfaces for any features I wanted to integrate so that I could add third party elements and still provide a unified interface. This had an unintended good effect: it helped to isolate poorly designed 3rd party interfaces (which at compile time created a lot of warnings). Thus I learnt I could use third party libraries provided I designed an interface to isolate from my code (this also simplified testing third party libraries since all I had to do was implement the library using my own designed interface). I ended up using Audiere and IrrKlang libraries for the sound system part of the engine.

3 - Side development

This was a direct consequence from both previous problems. When I did not have existing code, I first realized a simple small application as a proof of concept. When the proof of concept was ready, I kept rewriting it until I was satisfied with its design and architecture. Once done, I refactored the code for integration within the engine. Some development went right (subsumption system), some went wrong (file system), and one went utterly wrong (sound system). I spent a lot of time rewriting the same kind of code over and over when I could have designed an interface and be done with it (which I ended up doing for the sound system to link with a 3rd party library).

4 - Cross-platform?

Cross-platform development is demanding and there is little information on the Net. Moreover, there was one obstacle I did not have time to overcome: to test the engine on a linux box. I used virtual machines to create linux boxes but I encountered many other problems like learning how to properly configure a linux system (moreover on a virtualized configuration). As a hobbyist, I did not have the time to both learn the linux side and develop the engine. So I used whatever information I had (taking care of endianness, strictly defining engine types, multiple header inclusion depending on preprocessor directive, …). Still, even though the code is very clean and follows my home-made rules for cross-platorm, it never was tested on a linux configuration.

5 - Planning

I had originally planned to develop this engine during one year as it seemed realistic. However, I forgot to include editors in my planning, which were sorely needed for creating resources for my samples, and of course additional features were not even planned at the beginning (more on the spot when I decided I needed that feature). Moreover, I had not taken into account the extensive debugging for cross engine bugs (e.g. when some state changes under OpenGL pollute the rendering of some scene nodes).

It took two years to develop the engine due to all the previous faults. Still that was quite tight since I was a hobbyist married with a kid and with a day-time job. But I did not want to quit before all the planned features were implemented and the engine stabilized. I am however glad that all added features were interesting in their own since I kept learning new areas I was not familiar with. Had it not been the case, I would have had a big morale problem. Managing ones morale is essential to progress in such a project.

6 - Personal objectives

Since I added new features to the list and created editors, this meant that the project was beginning to take a life of its own: I understood that I would only program to maintain and evolve an engine (interesting since you do learn a lot of things, mildly to not rewarding if you do not take care of your morale and if nobody use the engine). I had to choose between developing this engine or write small games and small code projects using more third party libraries.

Conclusion

I wanted to learn developing a cross-platform engine with no external dependencies. The goal is partly achieved: I learned how to develop, test, stabilize, maintain and document an engine. I also have completely implemented the original feature list. The engine however is not cross-platform until tested on a Linux box.

When comparing with the list of engines at Devmaster.net, I found out that it was quite a nice little engine that could be evolved and satisfy a few basic needs. Still, during the two years of my engine development:

other engines also improved and are now more than sufficient to satisfy the needs I had.
I developed all the necessary skills to understand and adapt these engines to my needs.
New hardware architecture appeared (dual core 64-bit machines, and more to come). Even though I took some steps to ensure no side effect with ulterior compilation on 64-bit platforms, I cannot guarantee this. In fact this means more tests to do on different configurations and I cannot virtualize to insure the stability of my engine on 64-bit architectures.

Was it a waste of time? Definitely not. The knowledge I gained was invaluable (understanding quaternion, refactoring code, scenegraph optimization…).

Can a lone programmer (even a hobbyist one) succeed writing an engine? Yes, provided she has previous succesful programming experience, she plans carefully and focuses on features that are within her reach. Success in realizing the plan brings enlightenment.

Discuss this article in the forums

Date this article was posted to GameDev.net: 10/8/2008
(Note that this date does not necessarily correspond to the date the article was written)

See Also:
Programming