My brain really wants to crawl out of my skull and go... someplace else.


My VS2005 task list nicely summarizes the situation:




This is going to be one heck of a massive undertaking, and the sheer power of it is just overwhelming. On the one hand, I'm rewriting technology that exists already, which means half the hard part is done - I know the functionality that has to be supported, I know how that functionality works, and I know what it takes to implement it. Even still, rewriting tens of thousands of lines of complex engine code is no cakewalk.

There are some deeply profound implications to making this kind of change to a very old codebase. A great example is the use of fixed-point math routines throughout the pipeline. Fixed-point was a good idea when the first foundation code was written - 12 years ago. In modern times, it's half as fast as floating point support, and requires a lot of MultiplyFixed() type crap instead of just using a * b - because, when the engine was originally written, C++ was pre-98-standard and not really a choice for use in a commercial game engine. That means no operator overloading, which means ugly code.

Rewriting the engine means moving out of fixed-point crap; but the problem is, a huge amount of code is written to assume fixed-point data. The net result is that the bulk of the game's 400,000-line codebase is affected by having floating-point data in the 3D engine core. Some of that can be mitigated by taking advantage of existing interfaces, where we can just do case-by-case conversions of data between fixed and floating point. But it's still going to force us to rewrite close to 70,000 lines of code to handle the data format change.

Another deeply pervasive problem is memory allocation. We have a reasonably decent allocator that does things like tag what section of the game is allocating memory, checks for leaks, and so on. The problem is it's written in the C style of allocation - malloc()/free() - which means that the vast bulk of the engine does a lot of manual memory management. That will all go away with proper use of STL containers, RAII, and some other tricks. The amount of dynamic allocation we do per frame is utterly incredible right now.


The work is more than worth it, though: it means that we no longer have to do 16 fixed-to-double conversions for every single transformation matrix on every single chunk of geometry in the entire game... once a frame. Maybe we can also pull out the shader-parameter code that's doing dozens of string comparisons per geometry chunk per frame.

I'm really starting to understand why the game is so amazingly CPU-bound, even with the outstanding graphics. It's a testament to the enormous talent and perserverance of the team that we've even kept the thing running this long.



Another sticky point comes up here. Since we are effectively reconstructing the entire scenegraph system and internal geometry storage/handling mechanisms, the existing D3D code will cease to work - it relies heavily on the old scenegraph design (such as it was) to know what data to push to the hardware. So we can't actually see the results of this engine rebuild until the render logic is retooled - except the programmer who knows the rendering code is already drowned in work, so that may be a while. In the meantime, we can't use the new scenegraph to get visible results without maintaining a parallel copy of the data in the old scenegraph - which utterly negates the point of doing the new one in the first place. So we have a sort of mutual-dependency thing going on, and the net result will be a few weeks of total blind work.



On the positive side, we now have an official codename for the engine: Phoenix. I made a joking reference to taking the existing engine and "burning it to the ground." From the ashes rises a new and much nicer engine... et voila, cheesy pun.

Hopefully I'll actually be able to produce something that is an improvement and not a net loss [smile]