|
My brain is built of paths and slides and ladders and lasers and I have invited all of you to enter its pavilion. My brain, as you enter, will smell of tangerines and brand-new running shoes.
  A Tale of Debuggery | 
Posted - 3/27/2007 3:43:42 PM | This is a story.
It doesn't have any guns, explosions, or crazy futuristic technology. There is no romance, no beautiful woman to be saved at the end, and no vague but stirring innuendo. There is no deep, insightful commentary on humanity or any of the goings-on of human beings.
There is, however, an interesting program bug; a brief glimpse into the guts of a commercial 3D game engine; some only-slightly censored "hot action"; and a shocking moral that will leave you breathless with introspection and personal revelation. I may also be making some of that up... but you'll have to read on to find out which bits aren't true!
4:00 AM, give or take.
Lately, I've been burning the proverbial candle, and not just at both ends; I gave up on that a couple of days ago and just chucked the whole thing into the fire. As part of some major code changes, I've checked in some code that, unfortunately, broke the build - shame, shame, I know.
I can't fix the build because the broken code file was locked for edits by another team member[1]. What with time zones and all, I won't be able to sort things out until about 8AM, optimistically speaking. So I'm going to bed, and getting up again at 7:30. Ordinarily, I would just sleep in and maintain a bit of my (admittedly fragile) sanity, but I broke the build... no good being out of contact for a whole day while nobody else can get work done.
8:20 AM... approximately.
We got things sorted out and the build is now fixed. I'm now thoroughly loaded with caffiene, just to stay conscious, so I figure I may as well keep working for a while.
9:03:08 AM
Another team member drops me an IM, reporting weird problems in the newly updated code.
Team member: if I get an exception I can't see the error message until I minimise the game window, but when I do that the rendering code crashes out losing me my message window and context
Huh. That's odd.
We spend some time talking out the details of the scenario, and then the phone rings. Bug is on hold.
10:10:15 AM
I check in what I think is a fix for what I think is the problem. Turns out I'm completely wrong, and the fix actually makes things worse by masking some of the symptoms. We talk over the scenario again for a few minutes, but something else comes up, and my teammate has to fix something else unrelated.
11:34:40 AM
Back on the case. We get a better handle on the problem. Part of the game database is failing to load, which is causing a NULL pointer to be passed around in the game logic. When it is finally dereferenced, the game crashes (as it should). I set up some hack changes to try and reproduce this problem.
11:48:25 AM
I've reproduced the problem easily... but it's become clear that the problem I've created is not the real problem. There are actually two exceptions floating around - one C++ exception from the failing loader code, and an SEH exception from the NULL pointer dereference.
What's bizarre is that one exception is thrown; there's a debugging mechanism that pops up a modal MessageBox() when the exception object is constructed. This lets us immediately step the debugger into the code where the exception was thrown, before the stack unwinds and any catch handlers are invoked - which is a particularly useful tool.
Oddly, when the message box for the debug option pops up, notifying of the exception, code appears to still be running. We're still building a single-threaded branch of the project at this point, so the behaviour is completely unexpected.
What's worse, this only occurs when the game is running fullscreen, and the user has to Alt+Tab away from the fullscreen window to see the message. My original fix was to just hide the DirectX window when the message is displayed, then restore it afterwards; this is technically something we want to happen anyways, but it makes the buggy exception stuff disappear. Running in windowed mode also causes the problem to go away.
11:58:06 AM
I decide to try and rig up a test case where I can see the call stack when the second, NULL-pointer exception is thrown.
11:58:24 AM
The dreaded word heisenbug is invoked, multiple times. Spines in the immediate area are afflicted by a sudden onset of chills.
11:59:19 AM
I successfully capture a call stack.
Unfortunately, the interesting bit is total crap, because I'm missing some debug symbols for some reason. We compare thoughts on the stack, noticing some weird voodoo going on in d3d9d.dll.
1:15 PM, roughly, thereabouts, etc.
After extensive buggering around with some Visual Studio settings (apparently, installing SP1 makes your symbol server settings go away magically) and downloading a few hundred MB of symbols from Microsoft, I'm back in business.
I recapture the call stack, and hit pay dirt:
Can you spot the source of the problem? Go ahead - give it a try before you read the answer.
You didn't try, you filthy cheater... you're just being lazy and skipping down to the good stuff.
Really, seriously - give it a shot. It's good insight into just how complex programming can be.
OK... either you've tried to figure it out now, or you're incorrigibly lazy. So here's the answer:
The bottom line is that when Direct3D runs in exclusive mode, it performs a window subclass on your main drawing window, to trap certain messages. Our code, when in fullscreen, was triggering this behaviour. The problem is, MessageBox() essentially runs its own little internal message pump loop, to ensure the box is modal.
Unfortunately, the process of Alt+Tab'ing out of the game into the debugger tripped a message which got pumped, by MessageBox(), to the main game window; this is then intercepted by the DXUT framework, which calls our rendering code.
The renderer isn't ready to render, because the game loading isn't done; the game loading, if you'll remember, threw an exception, which, thanks to our debugging mechanism, popped up the message box. Kaboom - nasty re-entrancy problem, and a mysterious null pointer exception that occurred when it "shouldn't have been possible."
The Fix
The fix was trivial: the exception debug mechanism which shows the message box was tweaked slightly. Now, before showing the message, it explicitly hides and deactivates the render window, which prevents the re-entrancy problem. Rendering is restored after the modal message box returns.
The Moral of the Story
There's a couple of interesting lessons to be learned here.
First of all, never trust a call stack if you don't have symbols for all of the code in the stack. When your debugger says "frames below here may be incorrect", it means it. We spent a few minutes on a wild-goose chase looking at DXUT's keyboard hooks because the debugger spit up some random gook for stack frames it couldn't understand; as it happened, it pointed us at some internal Windows routines that are normally used for invoking a keyboard hook callback. Oops! Fortunately, we didn't waste much time on this.
Second, keep your tools up to date, and know them well. It didn't take long to get the correct debugging symbols; most of the time was spent downloading files from Microsoft's inexplicably sluggish file servers. Also, when you download symbols for an SDK, make sure you get symbol files that match the SDK version you're actually using; trying to load the December 2006 SDK symbols when compiling against the August 2006 SDK doesn't work. If I'd been keeping my stuff up to date, I could have saved some time.
Third, when debugging, precise replication is essential - there is a fine line between reproducing an existing bug, and creating new ones. I spent a lot of time creating new bugs; it took a few bogus attempts before I reproduced the bug. Although thankfully this was hardly a critical, blocking issue, the time was still wasted. In a real emergency situation, getting the bug right the first time is very important.
Fourth, fixing symptoms is not enough; never stop debugging until you know the root cause. In this case, my first fix (which masked the symptoms) was actually the right fix - but that was pure luck. If I had stopped when the symptoms went away, I could have potentially left a very serious concurrency problem lurking the code, waiting to bite us at some critical point - like, say, three days before gold-master deadline. Understanding the root cause of the bug is always critical to ensuring that the fix will kill the bug dead, and keep it dead.
Finally, it is always valuable to understand the inner workings of code - even if that code isn't yours. Knowing how window procedures work and how modal dialogs are typically implemented was crucial to solving this problem. However, what was even more important was understanding window subclassing; that was the crucial element that led us to understand Direct3D's role in the bug, and discover the nature of the problem. It took bare seconds on Google to confirm that Direct3D subclasses your window in certain circumstances.
Re-entrancy problems are sneaky and devious, especially when dealing with apparently blocking operations like a modal message dialog, that you do not expect to cause re-entrancy into your code. Sometimes, they can even cause apparently modal dialogs to disappear and be replaced with null pointer errors from code that shouldn't have even been running.
3:55 PM
After several false starts, I finally finish this exceedingly captivating story, and submit it for publication on the Intertron.
And they all lived happily ever after to the end of their days.
[1] Yeah, I know, exclusive-mode is bad source control practice, blah blah. It's partly a cultural choice and partly just because it's easier than reworking a bunch of our pipeline; we mostly live with it and do a large portion of code changes locally and only check out files to merge changes. It just so happened that in this case things didn't work out that way. In any case, I'm not here to defend our source control practices, so bugger off 
| |
| Hey... YOU'RE DUMB! | |
Posted - 3/24/2007 10:27:54 PM | So, hands up: who likes to think of themselves as stupid?
Pause for crickets and tumbleweed...
Huh. Apparently, feeling dumb is not particularly popular. Who would have guessed?
Let's try a little experiment. I want you to imagine that you and I are in the same room, discussing something. You've just put forward some extensive effort and invested a lot of time and energy in some particular project. Let's say it's a really complex and intricate bit of code. I've just looked at your work, and I've taken a deep breath, about to express my reaction and opinion.
Now, get a good mental image of me, and imagine that I'm looking you in the eye. I want you to imagine this vividly - actually hear it - when I say the following words:
You're stupid.
How would you react to that? Be honest.
No, don't bullshit me. I said be honest. That's right. No lies. (If it helps, remember that this is all hypothetical and in your head - I won't really know. And unless you're screaming in rage and violently introducing your monitor to the floor, nobody else will know either.)
It's interesting (to me at least) how varied people's reactions are when they're confronted with some attack on their intelligence. It may not even be a real attack - even perceived slights can inspire all manner of powerful responses in people. Some will immediately enter denial-mode, offering all manner of excuses and rationalizations to back up their side. Some will become hostile, going on the attack, a sort of "you can't discredit me, I'll discredit you first!" attitude.
Even more interesting is how people tend to conflate intelligence with correctness. If someone says you're wrong, more often than not it will be interpreted as you're stupid. Try repeating the little thought experiment from earlier, except this time substitute the word "wrong" for the word "stupid." Do your feelings change? Do they change significantly?
On occasion, quite rarely, we run into an interesting sort of person: someone who understands the difference between being wrong and being dumb, even when it is they themselves who are wrong; someone who can be directly told, to their face, that they're wrong (or even stupid!) and not become hostile, angry, defensive, or generally unpleasant.
I am not that kind of person.
I don't have much going for me - no ravishing good looks, no Herculean physique, no superpowers or influential relatives, no particular ability to cook. The majority of my ego is rooted in the belief that I'm not stupid. This is, fundamentally, a pretty ridiculous state of affairs, because in point of fact, I am stupid.
There's an interesting principle that affects all forms of growth, and it holds just as much for muscle-building as it does for intellectual skills: if you do not stretch beyond capacity, there will be no growth.
You can't increase your muscle mass without increasing the work you make your muscles do. You can't increase your interpersonal skills without meeting new sorts of people. You can't increase your vertical jump without progressively ratcheting up the bar and forcing yourself to go higher.
You can't become a better programmer without stretching your abilities - working in a domain you do not know, working with a tool you do not know, working on harder or larger problems.
The principle itself shouldn't be news to anyone; it's cited often enough, and certainly the kinds of people liable to read a journal like this already know it. There is an important corollary to this principle, however, which is not mentioned so often: you can't stretch your abilities if you don't know your own limits.
But, you may object, that isn't really true; after all, I can just tackle some ridiculously hard project, knowing that it is well beyond my limits, even without knowing exactly where my limits are.
Technically, that's correct; but it isn't really a practical way to try to grow. Setting one's sights too high is never productive. It leads to disappointment, disillusionment, quitting, and maybe even severe injury. Pretty much everyone knows they can't lift a bus, but nobody tries to lift buses as a way to bulk up for swimsuit season.
The trick to growth isn't to stretch yourself so much that you rip apart under the stress and fling sticky bits of goo all over the room. The trick to growth is to stretch just beyond your limit - enough to be uncomfortable, enough to be noticed, but not enough to cause outright injury. We lift barbells, not buses.
The hardest part of knowing your own limitations is acknowledging them. I think that, to some extent, everyone knows deep down what they are and are not capable of. It may be buried way down there, but there's always the quiet voice of reason that says you're in over your head.
What's hard isn't getting that voice to talk. What's hard is listening to it.
I've been programming for something like 15 years now. One of my greatest regrets is that I haven't kept much of my older work; almost all of my original code is long gone. But I have preserved a few fragments, a few snippets of memory here and there.
There's one in particular which is a sort of paradoxical thing; it's immensely embarrassing and even painful to read, but simultaneously deeply valuable. It's one of my favorite reminders of my own stupidity.
Scribbled in pencil in an ancient spiral-bound notebook, there's a page with these words:
Quote:| Hi, I'm Mike, and I know pretty much everything about programming in QuickBasic. |
Every few years, as I shuffle my personal possessions around for various reasons, I come across that notebook. And every time I've opened it and read that horrible sentence, I've had pretty much the same reaction:
Ugh... shit! Did I really write that?
For the first few months of my hacking career, I knew my limits. I knew that there was a huge amount to computers, technology, programming, logic - and that I had barely scratched the surface. I knew it was hard, that I didn't really know what I was doing.
I knew I was stupid.
Somehow, over time, I lost that - to my own detriment. I learned fast, and wrote a lot of code. I can't say it was particularly good (hell, it was QuickBasic), but it wasn't terrible. And somehow, I let myself become convinced that I was no longer stupid.
The next time I really felt the burn of my own stupidity was when Windows 3.1 finally started to gain popularity, and I moved into the heady realm of Visual Basic. I had the cutting-edge, hot-off-the-presses 3.0 version, and it was incredible stuff. Windows was a totally different realm, and I once again was forced to confront the fact that I didn't know much.
Mercifully, the second time through the cycle, I didn't get quite as big-headed. Even once I'd produced (and sold) a couple of decent, small VB apps, I never did get to a point where I believed that "I know just about everything." I thought I was pretty hot shit, but I only had to look at that old notebook to be reminded of just how limited I really was.
The next big leap was C, and it was a bugger. VB had its pitfalls, especially in the old Win3.1 days, but it was nothing compared to C and 32-bit DOS extenders. (I feel old just writing that phrase...) Memory models? Near and far pointers? Interrupt vectors? VESA? What the hell is all this stuff? I was stupid - really, profoundly stupid, and it lasted for a while.
It's interesting to me that I never really did reach a point where I felt non-stupid in C. Certainly, I got comfortable and confident, and even became fairly productive. But there was always a nagging sense that the language - or, more accurately, the things you could do with the language - comprised a realm of knowledge much larger than I understood.
I'm not sure I can really credit myself with this newfound grasp on reality. I think the biggest influence was the Internet; I only really left VB for C around the time when the Internet was populous and accessible enough that I had easy access to other people's code. It never seemed very hard to find C code that blew my mind. In the early days, I'd only very rarely glimpsed anyone else's work, and I think that made it much easier to develop delusions of grandeur.
My early time in C lead inexorably to C++, of sorts. In those days, Win32 programming was done with direct API calls, and it was not pleasant. I remember when MFC finally became somewhat stable, and people started thinking of it as ready for "production code." It was such a huge breath of fresh air, and a glorious relief from direct Win32 API programming... but it required this new, weird, strange beast of a language, this C++.
Unfortunately, due in part to the rather unstandardized nature of C++ at the time, and due largely to the fact that MFC was really a pretty piss-poor C++ library, I developed a lot of misconceptions about the language. It seemed like C with some interesting, but largely annoying, extras bolted on. In the end, it was worth putting up with this weird C++ beast, if only because MFC was so much more convenient than the alternative.
So it was that I didn't really know C++ ... but I knew MFC, and that was enough to score a job. It was, ostensibly, a C++ programming job. The project wasn't the largest or most complicated thing I'd worked on by any means, but it had an interesting and annoying new dimension to it: maintenance. Unlike anything I'd done before in C or C++, I inherited a large block of code from someone else.
I quickly found out that there's a huge world of difference between working code and good code - good code adds the immense requirement of also being maintainable. The code I had was working, barely. It was not, by any stretch of the imagination, maintainable.
The solution, of course, was to rewrite it - but by that point I'd seen enough glimmerings of what real C++ should look like that I had my doubts. It wouldn't be enough to just rewrite it in the half-C slop style that I'd favored up until then; I figured, what the hell, I'm rewriting this application, I may as well go all the way and learn what real, good C++ is like.
And, once again, I was smacked in the face with a fact that had never really changed, but I'd somehow managed to ignore: I'm stupid.
In the intervening time since that project, I've learned a lot - about C++, about good software design in general, and about my own abilities. But I've once again gotten comfortable, slowly allowed myself to believe that maybe I'm not that dumb, that maybe, just maybe, there's a little bit of competence hiding somewhere in there.
One thing I've always sorely regretted about my own newbie-days in programming was the lack of mentors. With the (relatively) newfound tool of the Internet, it's suddenly possible to fill that role for a new group of beginners and learners, and I enjoy the opportunity to do so - partly because I like being helpful, and partly because it helps fuel the voice of the ego, which is continually saying you're not dumb.
The ego, however, is a lying son of a bitch - and that is how I came to post a really needlessly overcomplicated and generally crap solution to someone's question. Then, along came someone who is very much less stupid than me (thank the power of the Intertubes), and pointed out just how dumb my solution was.
D'oh!
There's a point to all this (and yes, I know I took my damn sweet time getting there). Once upon a time, back when I wrote really dumb and arrogant things in spiral-bound notebooks, my youthful ignorance would have prompted a certain response: "pfft, piss off, my answer's fine." I probably would have tried to invent some excuse as to why it was more robust, or made it more clear what was going on under the surface, or something - I don't know. (Thankfully, my bullshitting skills have found less and less use over the years, and seem to finally be atrophied to the point where it's not worth trying to lie anymore.)
At other times in my career, I would have responded in another way - by quietly disappearing and pretending nothing ever happened. The Internet allows a unique form of bailing out on an embarrassing situation. In real life, you have to at least endure the "walk of shame" and feel the burning stare of everyone on your back. On the Intertron, you can just vanish.
What really surprised me wasn't that I'm stupid. That knowledge lurks under the surface, and is never really too deeply buried. What surprised me was my first reaction - not anger, defensiveness, shame, or denial.
My first reaction was why didn't I think of the cleaner solution myself? There's nothing fundamentally unknown in Zahlman's improved code; I use the similar stuff all the time. I wrote a somewhat similar blurb of code earlier this very week. So why was my first response such a convoluted, horrible, idiotic, palaverous mess?
There's something important to learn from this, I think. I'm still stupid; I doubt that will ever change. But I've become, over the years, much more accepting of that fact. I finally know and acknowledge my own limitations, and as such I'm prepared to grow.
But clearly I'm not mindful enough of those limitations. If I really truly did know my limitations, I would have paused before posting the answer, to consider if it was really as good of an answer as I thought it was. It shouldn't have taken much consideration to realize that it was not, in fact, a good answer.
The final conclusion has pleasing Zen quality: I am stupid. I know this fact. I accept it and am, at least on some level, conscious of it.
But I'm still stupid about it.
| |
| God of War II dev diary | |
Posted - 3/22/2007 3:00:26 PM | Watch this video. Right now.
This is the most honest, brutally true-to-life, and transparent glimpse into the realities of the game development world that I've ever seen.
There's a vague sense of deja vu for me, watching this. The intense crunch pre-E3; the tense, half-pissed off meetings where everyone is picking stuff apart and getting their toes stepped on; broken and incomplete stuff in a demo, right at a critical moment; even the frank admission that game developers swear a lot.
I can't count the number of times I've heard things like "I thought we'd be much further along by now." The "it's a non-trivial matter" line cracked me up, because I say crap like that in our meetings all the time. Sleeping at the office, everyone being tense and stressed, that moment when you go from being human to being a code-producing, blood-filled robot that exists only to crunch it and hit deadline.
There's the glazed, dulled, exhausted look on the faces of the team; the pain of having stuff axed from demos - or, worse, from the final game - because it just wasn't ready to go; that final moment of utter relief and satisfaction when it's over and you've done it.
The closing comment is powerful, because ultimately we've all felt it: that knowledge that your career hangs in the balance, and that you will live or die in the industry by the quality of your work. When a studio tanks, it's gone; you don't get many chances to fuck up. And even when you're surrounded by extremely talented people, and you know you've done the best you can do, there's always that moment of doubt where you wonder if your best is really good enough.
I watch every "making of" documentaries from games that I can get ahold of, partly because they're rare, and partly because I find it fascinating to see how different - and, ultimately, how similar - other studios are.
They usually tend to be highly PR-polished, positive-note, touchy-feely crap. Frankly, most of them are lies. Game development is a real process with real people involved, and it's stressful. It isn't clean-cut and pretty. People aren't nice and polite and constantly happy. In all the stuff I've seen about deadline-stress and the desire to produce good work, nothing really has captured the truth of it like this video.
It's absolutely true that you reach a point where you want to "go home and punch something". It's completely realistic to have a big, important, high-stakes meeting where everyone is vaguely on edge and trying not to be blamed for whatever the big failure of the week is. The tension and half-rivalry and mutual encouragement and respect between team segments and individuals is completely true to life.
Watch the video. It's as close to experiencing a real game project as you can get without sleeping under a desk and eating cold pizza at 4AM.
Oh, and whoever has the model roller coaster toy, I want.
| |
| What have I become?! | |
Posted - 3/17/2007 7:38:30 AM | I have descended to the darkest depths of bachelor geekhood: I just stayed up all night on a Friday night, and my breakfast consists of Taco Bell Fire Sauce packets squeezed onto stale Fritos Scoops, because I haven't had a chance to go grocery shopping yet this week.
Rahh.
| |
| Wednesday, March 14, 2007 |
| What it's like to be a game programmer | |
Posted - 3/14/2007 2:35:17 AM | I was over in a thread and got all introspective and thinky and such. So I'm going to empty my brain before I go back to work.
I don't tell people what I do - at least, not if I can help it. Whenever I meet people, the conversation goes something like this:
Them: So... what do you do?
Me: Menial computer work.
If I'm lucky and they have some sense, they'll stop asking questions. Occasionally, though, you get one of those weird people who doesn't realize that "menial computer work" is code for "you don't want to know." What follows typically ends up like this:
Them: Oh, really? What do you do with computers?
Me: I make software.
Them: That sounds cool. What sort of software?
Me: *sigh* Video games.
Them: OH MY GOD THAT'S SO COOL WOW TELL ME ALL ABOUT IT WHAT'S IT LIKE DO YOU REALLY JUST PLAY GAMES ALL DAY BLAH BLAH BLAH
Me: I can't talk about it.
(This hypothetical exchange is only slightly exaggerated due to me being bitter and cynical. It often plays out just like that, except they're not really saying "blah blah blah" - that's just what I hear.)
Invariably, whenever this painful and damnable conversation occurs, I say the same thing: "It's not really as interesting as it sounds."
This isn't to say I don't like my job. I love what I do, and not too many days slip by without me reflecting on just how fortunate I am to have it. It's essentially perfect - I honestly can't complain.
But I have no way to relate to anyone. It isn't like retail or fast food or some other pedestrian career that everyone, in some way, has had contact with and therefore can understand. Nobody else writes games for a living. The experience of buying and playing a game is so utterly removed from the process of creating it that even avid gamers rarely have any concept of how their games come into being.
At one point, I imagined that GDNet would be different. After all, these are not the teeming, unwashed masses of Backwards, Georgia - we're all sort of into game creation here. Most of us have a pretty good idea of what it takes and how it works. So surely here I could rant and ramble about my daily work experiences, and find some understanding and interest - right?
Not so much.
The problem is, even to fellow game developers, there's really not much to say. I can't talk about how I solved Big Problem X because nobody else will ever have that problem - much of what I do is unique to the particular sort of games we produce. And, in the end, who really wants to read a five page story about how I stayed up all night hacking on an obscure glitch in the menu rendering code? Boring!
There is some interesting stuff... but between NDAs and general caginess, I'll never be able to talk about it until it's so outdated even archaeologists and ancient historians won't find it interesting. And most of that is just possibility - not even things we're totally committed to doing yet.
So what's it like being a game programmer?
I spend most of my time pretending I don't have a job, or just do something nondescript and vaguely forgettable like "insurance paperwork filing" or whatever. Every now and then I gloat about not having to punch the clock when my friends start whining about how they have to go to work in the morning. That's about it.
In public, I dread questions about employment, and do my best to manipulate conversations to avoid it. In some ways, I'm the guy who knows he has a really lame job, and is trying to keep it a secret. The problem is, it's a really cool job, and I just can't be stuffed explaining it in patient detail over and over again to people who think the Internet is Hotmail and MySpace. (Or, God forbid, to people who think the PS3 is a decent gaming product. But I'm not going to open that can-o-worm.)
Sometimes, in between trying to change the subject to politics and trying to telepathically convince people I do nothing all day, I go to work. And then, for a few glorious hours, I do the most fascinating and challenging and incredibly rewarding job I can imagine ever doing.
It's pretty much like being a secret superhero. You do all this really awesome stuff at night, while everyone sleeps, and then you spend the day trying to act like you sweep floors for a living.
Now, if you'll excuse me, the Mayor just called, and I really must be going.
| |
In locus hic, omnes res dementes sunt.
|
| S | M | T | W | T | F | S | | | | | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | | 15 | 16 | | 18 | 19 | 20 | 21 | | 23 | | 25 | 26 | | 28 | 29 | 30 | 31 | | | | | | | |
OPTIONS
Track this Journal
ARCHIVES
July, 2009
June, 2009
May, 2009
April, 2009
March, 2009
February, 2009
January, 2009
October, 2008
September, 2008
August, 2008
July, 2008
June, 2008
May, 2008
April, 2008
March, 2008
February, 2008
January, 2008
December, 2007
November, 2007
October, 2007
September, 2007
August, 2007
July, 2007
June, 2007
May, 2007
April, 2007
March, 2007
February, 2007
January, 2007
December, 2006
November, 2006
October, 2006
September, 2006
August, 2006
July, 2006
June, 2006
May, 2006
April, 2006
March, 2006
February, 2006
January, 2006
December, 2005
November, 2005
October, 2005
|
