A Hobbyist Game Engine Post Mortem
After spending several years developing mostly 2D/3D small applications in different languages on different platforms, I decided to create ex-nihilo, an object-oriented game engine with 3D graphics. My main reasons for creating my own engine were:
- I already had existing code samples (graphic display, skeleton based model, sprite, …)
- When comparing with existing engines at that time, I thought they were lacking simplicity compared to the kind of programs I wanted to write and did not offer direct support for sprites, fonts and/or user-defined GUI.
What Went Right
1 - Focus
I wanted a cross-platform game engine to realize 2D/3D turn based strategy games or simple RPG games like Final Fantasy 7. As such the preliminary features were:
- OpenGL for cross-platform graphics at least v1.2.
- Object-Oriented Design
- Scripting: simple to change on-the-fly combat resolution, to code spell/weapon effects and game events
- Import/Export a few formats: the Milkshape 3D modeller text and binary format + my own internal format for sprites and models, TGA, BMP and light length-encoded bitmaps.
- Simple physics (basic collision detection of Axis Aligned Boxes and spheres; basic navmesh handling).
- Hardware vertex transform and lighting (T&L)
- Basic Texturing
- Skeletal Animation
- Special Effects: Billboarding, Particle Systems, Simple Terrain Rendering (Hex based and square based),
- Fonts and GUI support
- Sound system (mod player).
- Embedded logging system.
- Basic Multithreading availability
2 - Planning
In my day-time job, I am used to writing software/process specifications and planning schedules. This made the process more natural to do as I knew how. Moreover, I knew what time I had available to me and planned in consequence. Even though my documents were less detailed than what I do at work (hey, that’s a hobby for me), I had defined goals to reach and found out it would require one year’s time to complete the project. I knew from the beginning it was no trivial task and was prepared for it.
Here is the simple planning I did:
- Design the architecture: 2 weeks
- Realize the core framework (interface for creating a blank Window + initialising OpenGL + basic helper functions and logging + basic multithreading interface): 3 weeks.
- Realize the basic core Renderer (frustum + renderer + math interface + Color and Light interfaces + texture manager): 6 weeks.
- Create the font engine (refactor bitmap and rendered font code, include default fonts, create font file format): 4 weeks.
- Create the basic Scenegraph (Scenegraph and basic volume generator): 4 weeks.
- Create the Mesh Model Node (refactor existing code and file loading, process animated and static models + model manager): 5 weeks
- Create the Sprite Model Node (refactor existing code and file loading, create billboarding code, process animated and static sprites + sprite manager): 3 weeks
- Create the Particle engine Node (create code, create interface): 3 weeks
- Create basic terrain rendering (refactor terrain generation code with hex and square cells, create file format): 2 weeks.
- Create the GUI interface (refactor existing code for buttons, dialog, text, progress bar, mouse widgets, create file formats for skinning and GUI loading): 5 weeks
- Create the scripting interface (refactor existing code for scripting): 2 weeks.
- Create the sound system (reuse DUMB and adapt with OpenAL): 10 weeks.
3 - Careful design
Before committing to development I took time to polish my engine architecture. Contrary to small scale projects, this one needed to be carefully thought. It took me about 3 weeks to define a satisfying UML diagram of the engine. It did pay in the long run as each new feature integration neither broke nor frankly stressed the design. So no mid-project code overhaul or architecture redesign was ever needed. This also helped refactoring previous code as I knew the interfaces and how it should interact with the engine. This was another proof of a sound design adapted to my needs.
4 - Previous experience
I was able to capitalize on all my previous experience. I had already written simple expert systems, a forth interpreter, 2D graphics renderers, 3D mesh viewers including skeletal animation, basic multithreaded resource loaders… all simple applications in different areas of the game engine I wanted to develop. This gave a great boost to the realization of this game engine and improved my refactoring skills. This confidence is also reflected from the beginning in the planning where I knew with great confidence it was not uncharted areas. Good for the morale to know before hand that tasks won’t be that difficult to do.
5 - Iterative development
To me, what was of utmost importance was that each new feature and optimisation added did not add bugs to the engine or degrade performance. So I wrote a few samples using the engine (and even editors) and each time I modified the code, I compiled the engine against all examples to check that they perform as they were designed to. All examples were also improved each time to include the latest added features: they doubled as engine tests. I also ensured the engine and the examples worked the same way on all three different computers I use:
- an IBM ThinkCentre 8429 with no added graphics card using Windows 2000.
- a home-made Pentium 4 2Ghz with an old GeForce graphics card using Windows XP Professional edition.
- a laptop AMD processor with an internal GeForce 780X graphics card using Windows XP Home edition.
6 - Engine documentation
As this was a really big scale project, I wanted to easily maintain code documentation for ease of reference in later developments using this engine. I learnt to use Doxygen and never went back. It really was simple to maintain documentation (instead of writing full descriptive documents under Word which were a pain to maintain). This also improved the reuse of the engine since I was able to develop samples two years after the beginnning of the engine development.
7 - Keeping track
I also manually kept a log of any changes I made to the engine in a simple text file. I also manually logged any bug encountered in another text file. Bug correction had higher priority than feature development. This log helped me to keep track of what I was doing and where I stopped (invaluable for a hobbyist programmer with little development time available) but also gave a moral boost when considering everything that was done and all steps ensuring the stability of the engine.
8 - Logging for debugging
When developing a multithreaded application, the only solution to trace a code or to time a code is through logging. I had explored the use of gprof for debugging purposes. Even though it is mandatory for single thread applications, I saw two drawbacks for multithread applications:
- often the debugger does not stop on the spot where the error occurred. Moreover the spot where it stops changes over some application execution.
- a few times the error was not stable when the debugger was on: the program could run normally without any bugs on one execution and crash on another.
What Went Wrong
1 - Creeping features
Even though I had carefully laid out my plans, I could not help adding a few more features. I added:
- To AI scripting system: basic subsumption and rule-based system.
- To texturing: multitexturing and bumpmapping.
- To rendering: render to texture interface.
- Faster font rendering.
- PNG bitmap loading.
- OFF mesh format processing.
- A XML file parser.
- A scenegraph file format.
- A curve generator using three different methods
2 - Not made here syndrome
Under the guise of learning things (or not having external dependancy), I discovered I was in fact writing code and reinventing the wheel. I wrote for example a file system which let me indifferently load a resource from a package or from a physical disc. Even though it worked, it was of lower quality than code available on the Net which was better realized, cross-platform and compatible with multithreading (like PhysicsFS by Ryan C. Gordon for file system abstraction).
After discovering this, I created interfaces for any features I wanted to integrate so that I could add third party elements and still provide a unified interface. This had an unintended good effect: it helped to isolate poorly designed 3rd party interfaces (which at compile time created a lot of warnings). Thus I learnt I could use third party libraries provided I designed an interface to isolate from my code (this also simplified testing third party libraries since all I had to do was implement the library using my own designed interface). I ended up using Audiere and IrrKlang libraries for the sound system part of the engine.
3 - Side development
This was a direct consequence from both previous problems. When I did not have existing code, I first realized a simple small application as a proof of concept. When the proof of concept was ready, I kept rewriting it until I was satisfied with its design and architecture. Once done, I refactored the code for integration within the engine. Some development went right (subsumption system), some went wrong (file system), and one went utterly wrong (sound system). I spent a lot of time rewriting the same kind of code over and over when I could have designed an interface and be done with it (which I ended up doing for the sound system to link with a 3rd party library).
4 - Cross-platform?
Cross-platform development is demanding and there is little information on the Net. Moreover, there was one obstacle I did not have time to overcome: to test the engine on a linux box. I used virtual machines to create linux boxes but I encountered many other problems like learning how to properly configure a linux system (moreover on a virtualized configuration). As a hobbyist, I did not have the time to both learn the linux side and develop the engine. So I used whatever information I had (taking care of endianness, strictly defining engine types, multiple header inclusion depending on preprocessor directive, …). Still, even though the code is very clean and follows my home-made rules for cross-platorm, it never was tested on a linux configuration.
5 - Planning
I had originally planned to develop this engine during one year as it seemed realistic. However, I forgot to include editors in my planning, which were sorely needed for creating resources for my samples, and of course additional features were not even planned at the beginning (more on the spot when I decided I needed that feature). Moreover, I had not taken into account the extensive debugging for cross engine bugs (e.g. when some state changes under OpenGL pollute the rendering of some scene nodes).
It took two years to develop the engine due to all the previous faults. Still that was quite tight since I was a hobbyist married with a kid and with a day-time job. But I did not want to quit before all the planned features were implemented and the engine stabilized. I am however glad that all added features were interesting in their own since I kept learning new areas I was not familiar with. Had it not been the case, I would have had a big morale problem. Managing ones morale is essential to progress in such a project.
6 - Personal objectives
Since I added new features to the list and created editors, this meant that the project was beginning to take a life of its own: I understood that I would only program to maintain and evolve an engine (interesting since you do learn a lot of things, mildly to not rewarding if you do not take care of your morale and if nobody use the engine). I had to choose between developing this engine or write small games and small code projects using more third party libraries.
I wanted to learn developing a cross-platform engine with no external dependencies. The goal is partly achieved: I learned how to develop, test, stabilize, maintain and document an engine. I also have completely implemented the original feature list. The engine however is not cross-platform until tested on a Linux box.
When comparing with the list of engines at Devmaster.net, I found out that it was quite a nice little engine that could be evolved and satisfy a few basic needs. Still, during the two years of my engine development:
- other engines also improved and are now more than sufficient to satisfy the needs I had.
- I developed all the necessary skills to understand and adapt these engines to my needs.
- New hardware architecture appeared (dual core 64-bit machines, and more to come). Even though I took some steps to ensure no side effect with ulterior compilation on 64-bit platforms, I cannot guarantee this. In fact this means more tests to do on different configurations and I cannot virtualize to insure the stability of my engine on 64-bit architectures.