Jump to content

  • Log In with Google      Sign In   
  • Create Account

The Bag of Holding

I survived! Sort of.

Posted by , 02 June 2006 - - - - - - · 123 views

This afternoon I packed up and walked out of the Egosoft office for the last time (well, at least on this trip). I now face a weekend in Germany before flying home Monday morning.

It's been an unbelievable two weeks. In some ways, all manner of things got accomplished - design decisions, planning, inventing some really cool potential features that are quite exciting (and some of them are long overdue). In others, though, it's been a perpetual pileup of more and more Stuff that has to get done.

I think Daerax made a remark about my task list screenshot and its scariness. I found that kind of funny - because those 24 tasks were for a single code file. Globally, I have roughly 100 TODO comments assigned to me, which doesn't include the dozens of task entries in the JIRA system, or the on-paper plans which are still more massive. I'd make a collage of the hugeness of work, but unfortunately most of it can't be discussed publicly.

Despite the pressure and incredible weight now riding on my shoulders, this has been an awesome experience. Hacking around on code until the late hours, mobbing local restaurants at odd times, gaming in the office as "research" (we were particularly interested in analyzing Half Life Episode 1 today), instant espresso machines, blaring bizarre music at each other...

This is really pretty much the idyllic environment that I always pictured when I dreamed about getting into the games industry. The only real complaint I have is that everyone has a heavy German accent and lives too damn far away from me, but that's really pretty trivial, all things considered [smile]

To be honest, I wasn't sure how this would go, and in some ways kind of dreaded it. A while ago, all the pressure from everyone on the team to move over here to Germany was really utterly unappealing. Now, though... it's starting to be tempting. Living in another country is still a royally complex pain in the arse, and I'm not sold on the concept just yet, but... it does sound nice. The real killer excuse for me at the moment is the weather - I'd die if I had to live in German winters.

I'll certainly be looking to create excuses for more trips over here, though [grin]

The Epoch project has pretty much gotten buried under the avalanche of other stuff I've been doing. Most nights I haven't even had the energy to check my email, let alone think about language design issues. I'm looking forward to getting that moving again when I get home, but for now it's nice to just ignore it and not worry about yet another drain on my time.

This engine redesign project is intense. I'm under huge pressure because pretty much the entire game project rides on this getting done. The technical challenges of building a 3D engine are immense enough on their own; but in my case, they're compounded literally a hundredfold by the necessity of integrating with a large established codebase.

In some ways working inside an existing project is nice, because the problem space is clearly delineated: I know exactly what features have to be duplicated; many of the mundane issues have been worked out and solved; and the overall structure provides strong hints as to how the various interfaces need to be designed. At the same time, though, it's a complete and total bitch. Every tiny little nuance of logic, sequencing, or whatever could either be a tremendously vital bit that keeps things working correctly, or it might just as likely be a completely arbitrary thing - or even buggy. So determining what aspects of the system must be mimicked by the new design - and which ones have to be replaced - is a daunting task.

I won't be able to do much in the way of specifics, but I'm strongly considering writing up some of the design solutions I'm using when the dust settles. I see a lot of problems coming up in the forums that are painfully familiar, and yet trying to explain good solutions in the context of someone's existing engine is extremely hard, especially when I can't look at their code. I know we already have a decent chunk of articles contemplating engine design, but I think it might be kind of cool to describe how a "real world" engine was designed and implemented, from the ground up.

One thing I virtually never see done "right" in such articles is a description of the design decisions, which frankly are the only really important things to be teaching. The nuances of a system are fairly trivial by comparison - the hard part of the work is analyzing requirements, constraints, and possibilities, and deciding between competing alternatives. So I'd really like to cover 3D engine architecture from the point of view of making decisions to fit into an existing game's structure.

Technology is really just amazing. Sitting in a hotel room on an open wifi relay and surfing the web is freakin cool in a way that is just utterly impossible to describe to anyone who hasn't done it. At the moment, I'm at a loss for nice poetic phrases to really capture the mind boggling wonder of modern electronics.

I'm on some kind of emotional high right now, I guess largely from the huge stimulating effect of getting assigned the jobs I've been given. Well, that and making some really rather funny inside jokes all day. The massive doses of caffiene and sugar are probably contributing in their own small way as well.

It is good to be here. I think I'll do some heavy gaming tonight in a way I haven't done for far too long. It is indeed good to be a gamer.

But it's even better to be a game developer.

Insert Scream of Terror

Posted by , 01 June 2006 - - - - - - · 92 views

My brain really wants to crawl out of my skull and go... someplace else.

My VS2005 task list nicely summarizes the situation:

TODO: write an entire 3D engine

This is going to be one heck of a massive undertaking, and the sheer power of it is just overwhelming. On the one hand, I'm rewriting technology that exists already, which means half the hard part is done - I know the functionality that has to be supported, I know how that functionality works, and I know what it takes to implement it. Even still, rewriting tens of thousands of lines of complex engine code is no cakewalk.

There are some deeply profound implications to making this kind of change to a very old codebase. A great example is the use of fixed-point math routines throughout the pipeline. Fixed-point was a good idea when the first foundation code was written - 12 years ago. In modern times, it's half as fast as floating point support, and requires a lot of MultiplyFixed() type crap instead of just using a * b - because, when the engine was originally written, C++ was pre-98-standard and not really a choice for use in a commercial game engine. That means no operator overloading, which means ugly code.

Rewriting the engine means moving out of fixed-point crap; but the problem is, a huge amount of code is written to assume fixed-point data. The net result is that the bulk of the game's 400,000-line codebase is affected by having floating-point data in the 3D engine core. Some of that can be mitigated by taking advantage of existing interfaces, where we can just do case-by-case conversions of data between fixed and floating point. But it's still going to force us to rewrite close to 70,000 lines of code to handle the data format change.

Another deeply pervasive problem is memory allocation. We have a reasonably decent allocator that does things like tag what section of the game is allocating memory, checks for leaks, and so on. The problem is it's written in the C style of allocation - malloc()/free() - which means that the vast bulk of the engine does a lot of manual memory management. That will all go away with proper use of STL containers, RAII, and some other tricks. The amount of dynamic allocation we do per frame is utterly incredible right now.

The work is more than worth it, though: it means that we no longer have to do 16 fixed-to-double conversions for every single transformation matrix on every single chunk of geometry in the entire game... once a frame. Maybe we can also pull out the shader-parameter code that's doing dozens of string comparisons per geometry chunk per frame.

I'm really starting to understand why the game is so amazingly CPU-bound, even with the outstanding graphics. It's a testament to the enormous talent and perserverance of the team that we've even kept the thing running this long.

Another sticky point comes up here. Since we are effectively reconstructing the entire scenegraph system and internal geometry storage/handling mechanisms, the existing D3D code will cease to work - it relies heavily on the old scenegraph design (such as it was) to know what data to push to the hardware. So we can't actually see the results of this engine rebuild until the render logic is retooled - except the programmer who knows the rendering code is already drowned in work, so that may be a while. In the meantime, we can't use the new scenegraph to get visible results without maintaining a parallel copy of the data in the old scenegraph - which utterly negates the point of doing the new one in the first place. So we have a sort of mutual-dependency thing going on, and the net result will be a few weeks of total blind work.

On the positive side, we now have an official codename for the engine: Phoenix. I made a joking reference to taking the existing engine and "burning it to the ground." From the ashes rises a new and much nicer engine... et voila, cheesy pun.

Hopefully I'll actually be able to produce something that is an improvement and not a net loss [smile]

Brain asplode

Posted by , 01 June 2006 - - - - - - · 92 views

I'm literally unable to find words to describe this past week.

Stuff has been insane. The past three days were spent in highly intense meetings, with two other team members coming in from the UK for discussions and design sessions. We've been starting around 11 every morning, and often carry on with discussions until 2 or 3 AM. Just getting through a day has been powerfully exhausting, and I've repeatedly stumbled back into my hotel room in the early morning hours and just collapsed with no interest whatsoever in keeping up with email or the forums or whatever.

Despite the taxing pace, though, it's been a huge amount of fun. Every now and then something will come up and the discussion will get fairly passionate, but usually we all see fairly eye-to-eye on the major issues. There's a lot of joking and swapping of silly anecdotes and general good times. So it's been enjoyable and powerfully productive, but I'm totally knackered [smile]

We've got some really exciting stuff in the works for this next project. I can't give any specifics yet, especially since a lot of it is uncertain and maybe not even possible, but I can say that if we get even a fraction of our plans accomplished, the gameplay is going to be transformed - and vastly for the better. There are some seriously radical changes in the pipe for a few years in the future, as well - things that I think pretty much everyone will be excited to see happen in the series.

The catch to this is that our current production pipeline and methods are not up to the task. During the production of X3, a fair number of problems with our methodologies and tools came to light, and we're now in a position where we need to fix those issues. What makes it tricky is that many of the issues are deeply rooted in the philosophy and design of the existing code.

Our big goal is to deliver more - and better - content, of many different types. We're a very small team, especially in today's world of huge production studios, and to be bluntly honest we just don't have the manpower to do huge amounts of content unless our pipeline becomes more efficient. We're literally at the point where shaving seconds off of some tasks can have profound effects on the amount and quality of content that we can produce. So in order to maximize our content production ability, we have to really take a hard look at the pipeline itself and make it as robust and efficient as possible.

One specific outcome of this general situation is that we've found some very deep structural problems with the way our 3D engine handles animated data. It works great for certain classes of geometry and animation, but is simply too limited to properly handle all the nifty cool technologies we're working on. This means that roughly 90% of the 3D engine will need to get replaced entirely, built from scratch to help eliminate some legacy code dependencies. Effectively the only thing that will stay the same is our (quite good) Direct3D access layer and shader system; all geometry handling and animation handling will be rebuilt from the ground up. About 30% of the game's codebase will effectively be replaced by this restructuring.

I'm particularly interested in that because it's fallen to me to do the rewrite, at least of the core logic; there are already other programmers on the team working on retooling the art pipeline and much of the "client" code that uses the engine's services. But rebuilding the core is still a huge job - about 20,000 raw lines of code are getting replaced, and the new core has to be absolutely rock-solid so that the rest of the systems can still use it.

In some ways that's a huge thing to pile on top of my cutscene system job, but in other ways this is the only way things will ever get done. By rebuilding the system now, I can design it to make the cutscene stuff pretty easy to add. So even though it's a massive task, it actually gives me a net savings overall. Even better, it's going to radically improve our art production efficiency, which will quite likely have some profound effects on the final game [smile]

So I may be more or less incommunicado for a while. There's a huge pile of stuff to be done, but it's going to be so very worth it - even just the obvious benefits are already massive, and there will be all kinds of really cool potential that we haven't even thought of yet. It's a little early to make any real projections, but the daydreaming half of my ego can see these changes really making us a power player in the industry as a whole. We already have an amazingly capable team; we're effectively making that team several times more efficient and giving them tools to do far more than has ever been possible.

The outcome should be quite exciting to watch.

Mmmm.... beer.

Posted by , 27 May 2006 - - - - - - · 67 views

Well, I'll say one thing... us Germans sure do know how to make a beer.

Tonight's brew of choice was a nice Erdinger hefeweisse. Light, smooth, and very gentle. It's a good beer, although slightly too light for my personal taste. One thing I've noticed is that the imported shit we get in the U.S. is generally very little like the actual local brew, which is invariably much better fare. Such is life as an American: doomed to drink the leftovers of the rest of the world, who all actually have good taste.

Anyone know of any other good brews I should try and snag while I'm here?

Artist Ex Machina

Posted by , 25 May 2006 - - - - - - · 153 views

The first word that comes to mind is "enlightening."

Actually, the first thing that comes to mind is rather a phrase, and the words typically average four letters. I won't repeat it here, as I suspect even the veteran sailors among us would be made slightly uncomfortable.

(Now I'm out of hyperbole and melodrama, so I'll just tell it straight. OK, maybe with a little bit of exaggeration. You know, screw it - 99% of what follows is probably total lies. But at least they'll hopefully be entertaining lies.)

So this cutscene system is getting larger. And by "getting larger" I mean that I am more or less resigned to dwelling in a cave for several millenia, doing nothing but writing code and subsisting on recycled guano, just to get the outline of the functionality done. It should only take a couple of lesser eternities to actually produce a working alpha.

It isn't really as bad as I'm trying to make it sound, but it is rather severe. Originally, I'd understood the system as having two main parts: a sort of generic rendering service that draws pretty movie clips on the screen, and a semi-intelligent decision system that picks out different interesting things to draw, and feeds them into the rendering chain. I arrived here at the company office proud of my accomplishment (I'd finished a viewable prototype of the render chain, loading a dynamic scene from XML files) and was ready for my parade in the streets, complete with fountains of beer and numerous women with... shall we say... highly friendly dispositions. Well, I would have settled for a "cool, you're on schedule, that's great. What's next?"

Instead you'd think I came in and announced that I was going to kill everyone by suffocating them in my rectum. There was that kind of awkward silence and shuffling-of-feet going on. Although, to be fair, I was asked to give the presentation rather suddenly. I'd dismantled the actual live-demo part in the airport during a layover on the trip over, so all I really had to show was stuff that's been in the project wiki for almost two months, and the demo XML file. I'd also been roughly awakened from my jetlag-induced nap no more than an hour before. I'm not entirely sure if the response was due to my presentation's utter lack of content, or if I inadvertently used a term like "schweinhünd" along the way. I may never find out.

In any case, as it happens, the expectations for what this system will do have grown quite a lot since the last team discussion we had (at least, the last one I was involved in). The "semi-intelligent" side of things - the part that picks out cool stuff to show on the screen - has inflated a bit. I would have been ready to handle, say, Data from Star Trek, or perhaps even the HAL 9000. Unfortunately, things are slightly out of hand. Suffice it to say that by the time this thing works according to the current spec, it will set new standards for measuring just how smart something is. In fact, if you randomly pick a sequence of 10 lines of code from the (hypothetical) finished code, those ten lines will - by themselves - be smarter than Chuck Norris. I think it is now clear just how far things have gone.

The system now is going to need at least three tiers on top of the existing (and basically done) rendering framework. The first is a sort of cameraman that is responsible for composing individual shots, given a list of subjects (a ship, a planet, and that thing over there that's blowing up) and some priorities (explosions cool, planets pretty, ship not so important just now). This will be challenging enough simply because it requires a lot of logic that is typically approached subjectively by the artists - which is to say, there are no written rules for how to make a particular shot "look good."

It doesn't stop there, though! The next layer up is a sort of director, who will go over a scene and pick out groups of interesting subjects. The director then gives the subjects to the cameraman logic and says "make a nice shot showing this stuff over here." After the cameraman decides on a shot, he generates some parameters for the render pipeline, which then in turn boils things down into atomic operations in the actual 3D engine itself.

Overwhelmed yet? We're not done. The final layer is an evolution of the "event watcher" system that I'd originally thought was all the smartness needed by this thing. This top layer observes the entire universe and acts as a sort of agent for the player, picking out hotspots of "important things" and telling the director logic to show them.

To make this all the more interesting, each layer has to be accessbible directly without touching the upper layers. So we need the power to manually control a scene (which is now basically finished), or to manually control what shots are displayed, or to manually specify an environment in which the director should find interesting stuff to show.

Thankfully, I've got a very solid base design that will be able to handle all this data, and stacking the different systems will work out. It actually fits nicely into the horizontally-stratified abstraction/"mini dialect" concept that I've burbled about at great length. The only downside is that all the people interested in the system itself are also people who don't spend their time contemplating software architecture, so it's very hard to convince them that A) all the work I've done on the base rendering chain is actually necessary, and B) it's better to build this from the bottom up as a general system, instead of just hacking in a bunch of special-case code to finish the tech demo they want to see.

Interestingly, the artists themselves don't really see this as feasible for the most part. Most of the pressure is coming from Down On High to reduce the amount of work needed to generate cutscenes, plus of course the obvious bonus that once we have this technology we can use it all throughout the game for various cool things. The term "clever code" has been abused enough to make a two-dollar hooker feel positively loved. I've started privately referring to this entire escapade as the "Artist ex Machina." I deeply fear that the code may need to be more clever than I am, which makes it a bit hard to write - and debug. (There's some famous quote about that somewhere but I'm too lazy to dig it up.)

All of this stuff might be possible, maybe, but it's going to take a heck of a lot of work, and even after all that, we have no guarantees that the results will look good. So no pressure on me, really; we don't have an entire product design riding on this, or anything. That's definitely not a big load to carry out of bed every morning. Nope.

But then again, hell, this is what I live for. I was perpetually pissed at my last job because I never really had any major challenges. I'm just getting what I wished for... now I have to find out if that wish was a dumb mistake [smile]

There's plenty of other interesting little tidbits of team culture and personality coming to light, but I'll have to bore you with those at a later time, because right now I should probably be working [grin]


Posted by , 23 May 2006 - - - - - - · 110 views

OK, now this is just cool.

Here at the office they've got a little mini instant espresso machine thingy. You get a little pre-sealed, individually wrapped chunk of compressed coffee, stick it into a little slot, select your preferred strength, and poke a button - and in 30 seconds you have a nice little espresso. Preheats the water to all the magical temperatures, and all that stuff.

Apparently it can do proper cappucinos as well, but it's something of a black art to regulate all the temperature and stuff. Andreas, our 3D engine guy, is apparently the local master of the art, so I'll have to get some training before I leave.

And now back to work, with proper updates later.

Whee, jetlag is fun!

Posted by , 23 May 2006 - - - - - - · 93 views

Well, this has been a most disorienting 24 hours.

I arrived in Frankfurt yesterday (I'm pretty sure it was yesterday...) at about 6:30 AM, half an hour early. Let me tell you, these German folks really know how to run a passport control/customs line; I've been a few places and Germany has consistently (both times I've been here) been fast, efficient, and utterly painless. Didn't even have to fill out any forms. Pity that all the engineering skill in this country couldn't save the horrifically bad security design at the Frankfurt airport... it's an utter nightmare. If you've ever flown through there, you know what I mean. If you haven't... well, it's more or less indescribable. Just imagine getting mag-wanded and cavity searched every 30 paces or so and you have a rough idea of the situation.

I finally managed to escape the aiport and down to the lower levels where the train station hides. I really need to learn to read the language better, because I was about 10 seconds from missing my train when I realized that the huge destination it had displayed on it (Brussels) was in fact the endpoint, and my destination (Aachen) was very much on the route. Whoops. At least I made it on...

... for all the good it did me. I got to the Aachen train station in about an hour (damn, high-speed trains are fun!) and proceeded to find absolutely no one and nothing that I recognized, except for a McDonald's. Now, I was fairly clear that the arrangement had been for someone to meet me at the train station, so after hunting around for clues that I was in fact in the right city, and was in fact in a train station, I figured it was panic time.

Thankfully I had the sense to get the office's phone number before I left, and bungled through the payphone interface until I got ahold of someone and pleaded for rescue. (It took me a few dialing attempts of punching in the sequence of numbers and being scolded in German. I finally figured out that the "49" I had scribbled at the beginning of the number was, duh, not part of the number, but the country code. A German pay phone didn't want me making international calls to... Germany. Are we confused yet?)

After that relatively minor hiccup, I learned that the (admittedly crude) flight itinerary I'd sent over had listed my departure as the 22nd (which is true) and arrival as the 23rd (also true, due to the overnight flight) but nobody had noticed the "23rd" part. So apparently I was expected a day ago... whoops. It also turns out that there's a holiday this week so there's really only one solid day of work when everyone will be around.

I crashed at the hotel for a couple of hours, and then was awakened by a phone call and summoned back to the office. (Which was fine with me, since I was fairly bored at that point.) Unfortunately, there is no clock at all in the hotel room, and I'd left my laptop in the office. So when I woke up, I literally had no idea what time it was. That state more or less persisted until I came back to the hotel later that night and crashed again. However, this time I had the sense to bring my laptop back and rig up a little alarm clock mechanism, which proceeded to wake me up at midnight and 6 AM. I wish I could blame a mysterious technical gremlin for those problems, but they were literal times that I chose for no reason I can recall. It made sense at the time, I think.

I think the idea was to get up early, do a couple hours of work, and then hike over to the office (it's a pleasant walk through a nice little wooded area from the hotel) around 10 AM or so. Except I went back to sleep instead, thanks to the awesomeness that is a good duvet, and woke up just a few minutes ago.

Much to my shock (given the semi-rural-ish area the hotel is in), I discovered at some point during all that madness that there's an open wireless access point somewhere close by, so I can take the time to write long-winded and pointless journal entries without having to explain to anyone why I'm on GDNet and not writing code [grin]

Anyways, I think it's getting close to time to head out. I have to spend some time talking with the artists about some stuff, which I don't really remember clearly at the moment, and will have to describe properly later.

... but maybe I'll go back to sleep for a few weeks first. I need one of these at home...

Hit the ground running

Posted by , 23 May 2006 - - - - - - · 75 views

I'm here, and already busy as all get out.

More later, maybe [smile]

Mmmm... Deutschswaggen.

Posted by , 21 May 2006 - - - - - - · 63 views

Woo... now that is the way to crash after a crunch week! I slept pretty much all day today, so really all I have to do now is throw some stuff in a suitcase and burn some time until my flight leaves.

Apparently, Germany is cold, which I do not particularly like (being a tropically-blooded kind of guy). So I have to pack lots of warm clothes, which means lots of volume. I sure hope there's good laundry service at my hotel, because I'm definitely not going to be able to fit 2 weeks worth of warm clothes into my dinky little suitcase.

I more or less skived off work today, what with the sleeping and all - and hell, it's a weekend anyways. Once I get done packing I think I'll make a night of it and play some games.

Next stop, Germany... unless of course there's good free wifi in Charleston (layovers FTW!) in which case I'll make a quick update from there, just to annoy Daerax [grin]

On second thought, if GDNet continues to be this flaky, screw it. This is getting really damn annoying.

January 2017 »

1516 17 18192021