And now, something about work

Published March 14, 2008
Advertisement
Game Programming and Krystal Burgers1
Since something interesting is going on at work, I thought I'd share. Well, interesting and something I can discuss publicly, which is a rare combo.

The last couple of days have been spent in an intense bug hunting frenzy. Earlier this week, I was working on a small demo for an upcoming milestone, when a couple of memory bugs cropped up. What started out as a memory leak became a nasty chase through a convoluted inheritance hierarchy, dozens of misapplied templates, and some truly backwards code design.

Ironically, just at the end of last week, I'd signed off on that code as more or less usable, which is why I was doing a demo in the first place - I had vouched for the technology as solid enough to be showing off, albeit in a rudimentary form.

Apoch's n++th Law of Software Development:
You cannot truly judge the quality of a block of code until you must locate a bug in it.

Corollary
The longer you use a piece of code without having to hunt down a bug in it, the closer you can come to saying the code works correctly. However, this approach is asymptotic - you will never get there.


Fixing the Bugs
After a few hours of analysis, I determined that salvaging the code was out of the question. It would take less time and effort to rearchitect it and then copy/paste the functionality back in.

So that's what I've been doing lately. There's a huge chunk of work finished and ready for a version control commit, but a few major elements are still missing. As of right now there's actually less functionality working than when I set out to build the demo, so... bummer, I guess. That's the way things go.


Postmortem: Dissecting the Insects
So how did this situation come about in the first place? It's a simple and sadly common problem.

About a year ago, we had a newcomer to the team, who was interested in working on the engine and doing some game logic stuff. This particular technology was already written in another scripting language for an old engine, and needed to be ported to C++ for future use. The demand was minimal, so he was under no time pressure, and it seemed like a good project for a beginner.

As it turns out, the entire thing suffered badly as a result. He made a valiant effort, to be fair, but the result just wasn't up to par. The original programmer of the technology did some work to make it a bit better, but by and large it was a low-priority item, and thus languished in disrepair for months.

Worse still, the new guy disappeared after a few weeks, and nobody seems to know what happened - or, if they do, they're keeping quiet. It's unfortunate from a production point of view that a lot of time and code was wasted, but it's worse that a potential team member who had been trained (partially) on our workflow pipeline has been lost.


The Newbie Problem
This introduces a difficult question. Let's say we have a team of programmers, all of whom have an average skill level of X. Then a new guy joins up, with a skill level of Y, where Y is significantly lower than X.

It seems at first glance that no matter what J. Random Newbie does, the team suffers as a result. If he works on critical technology, chances are it will fail because it isn't up to the team's standards. If he works on something low priority, chances are it will crop up as a hidden cost when the item becomes high priority at some point in the future - and these surprises can bite you hard.

My personal opinion is that the only way to really do this is to let Mr. Newbie observe for a while, almost like a pair programming arrangement. Assign him to rotate between existing coders, and have them explain what they do, how and why they make their decisions, and so on.

It may seem like an annoyance and a drag on the veterans, but in my estimation, it will pay off immensely, for three reasons. First, the veteran's work is subjected to scrutiny - both his own and J. Random's - which is likely to increase its quality. Secondly, chances are the veteran would have to give up some time to slow his pace down and meet Mr. Newbie on his own level anyways; for example, if Newbie does Random Module Z, Veteran will have to integrate with Z at some point, and likely discover all kinds of things that need to be fixed or improved.

The third point, though, is the one I feel is most important: mentoring newbies in this manner creates veterans much faster than the old "sink or swim" method. It reduces turnover. It decreases stress, because everyone expects the training to be going on - it's all too easy for veterans to just expect the new guys to be up to speed, and get frustrated when that is not the case. Last, but definitely not least, it raises the overall quality of your code.

I'm not really in a position to institute this kind of policy at Egosoft, nor do we currently really have anyone who could realistically be put into a apprentice-type position; but next time the situation arises, I'm definitely going to be putting forward an argument for it.


Bipolar Update
I've worked almost solid since 6:30 this morning until now (about 1:30 AM), minus a couple of hours for meals and a brief nap. It's starting to take its toll on my motivation, but not my mental energy. That's a really good sign.

This is the first really strenuous push I've made at work in a very long time. It's refreshing to be back at a point where I can sprint when I need to. Time will tell if I end up crashing and burning afterwards, but so far none of the signs of a typical manic buildup/crash are manifesting. Of course, that might just be because I'm on so many drugs that all the symptoms are utterly masked.

In any case, it looks like I'm finally coming out of the long tunnel of bipolar disorder and starting to regain some semblance of normal functional life. Just don't judge my state of dysfunction by looking at my apartment - it's a wreck.


1 - Yes, Krystals. We don't get the Holy White Castle here in Atlanta, and although Krystal is a very distant - nigh-on-blasphemous - second place, it suffices to satisfy the occasional urgent crave. This is especially true during crunch time.
Previous Entry Ein Doppelpost
0 likes 6 comments

Comments

ukdm
My thinking has always been that some kind of mirror-tasking is the best way for newbies to get up to speed while keeping their motivation high. This is how I envision it working:

1. New programmer starts and spends a few days just understanding what everyone in the team does
2. When new tasks are assigned to the team veterans the newbie gets one of those tasks as well and the same time period to complete it
3. When the time is up the veteran's code and the newbies are both reviewed and the best version used

Now the veterans code is likely going to always win, but the newbie is motivated to try and do better and have their code used instead of the veteran. There is minimal risk as you have a backup of veteran code for every task the newbie carried out. Also, if they do manage to better the veteran you know they are picking up your pipeline. This method also has bug checking built in through the comparison of the code.

You can also repeat this for as long as necessary with enthusiasm levels remaining relatively high. If they don't then you probably don't want that programmer to stay anyway.

The two negatives in this equation are the extra time it takes to compare the code and possibly a veteran programmer with a bruised ego if their code is not chosen.
March 14, 2008 01:57 AM
Telastyn
I just recently started coding professionally. Not in gamedev, but with a large financial processing system. The things that helped me, and the team most wasn't writing any code, but tracking down low priority/annoyance, high difficulty bugs. It forced me to look/fiddle through code. It forced me to learn how/why things were done and get into the mindset of how things were done onsite. It forced me to ask questions of the senior devs, and learn what terminology to use, who owned what, how to ask good questions. It forced me to make sure I could build stuff and fiddle in the test environment.

And then once I found the bug I would generally have a really good idea how things worked around it and how to fix it in a style consistent with existing stuff.
March 14, 2008 07:58 AM
Ravuya
Have you guys considered regularly scheduled code reviews? Getting the newbie to go through his design or code in front of the group is probably helpful for all.
March 14, 2008 08:55 AM
Gaiiden
Quote:Original post by ApochPiQ
Worse still, the new guy disappeared after a few weeks, and nobody seems to know what happened - or, if they do, they're keeping quiet.

Oh, great. Once word starts getting out about the Game Development Secret Police dragging bad coders off in the dead of night on top of insane hours and crazy work pressures no one is going to want to work in this industry.

Way to keep it hush-hush there Mike. Good job.
March 14, 2008 10:04 PM
kevtimc
Quote:Original post by ApochPiQ
Time will tell if I end up crashing and burning afterwards, but so far none of the signs of a typical manic buildup/crash are manifesting.


You said you're type II; don't you mean hypomanic? (I'm just being dumb -- you don't really have to answer that [lol])

Moving right along...
March 14, 2008 10:34 PM
ApochPiQ
Quote:Original post by Ravuya
Have you guys considered regularly scheduled code reviews? Getting the newbie to go through his design or code in front of the group is probably helpful for all.


I've been pushing for it, along with one of the other guys, but the company culture just isn't quite ready yet. The team is still led by old-guard C hackers whose mentality is pretty much "if it works from the player's point of view, the code is fine."

Our tech lead especially is a terrific programmer in that style, but we still disagree about the importance of Total Cost of Ownership. From my view, TCO means that code reviews and maintaining everything at high-quality is crucial, because we are constantly needing to massage and adjust the code. The easier it is to refactor, for example, the faster we can work. However, from his point of view, doing refactoring is a waste of time, because we're constantly massaging and adjusting the code. See the dilemma? [smile]

So for now I largely just put up with it and get work done, since that's the Right Thing To Do, but I'm constantly on the lookout for persuasive arguments that might help shift things towards a more disciplined approach.


Quote:Original post by Gaiiden
Oh, great. Once word starts getting out about the Game Development Secret Police dragging bad coders off in the dead of night on top of insane hours and crazy work pressures no one is going to want to work in this industry.

Way to keep it hush-hush there Mike. Good job.


Hey, did I say "Secret Police"? No, you said it. So blame for that little slip-up is solely on your shoulders.

I'll ask the ninjas to make it fast and relatively painless, though - you've been a good servant.

I mean - that is to say - I know nothing about this "secret police" business. Please join the game industry, your life is not at risk. At the end of your career, there will be cake.


Quote:Original post by kevtimc
You said you're type II; don't you mean hypomanic? (I'm just being dumb -- you don't really have to answer that [lol])

Moving right along...


Heh, you're technically correct. However, in my experience, the vast majority of people don't know the difference between hypomania and true mania, so just calling it "manic" in the colloquial sense (and not the medical sense) is easier.
March 15, 2008 10:02 AM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Profile
Author
Advertisement
Advertisement