Tools: the unsung, underpaid heroes of everything

Published August 14, 2006
Advertisement
I want you to do something. This is something that virtually everyone can do. You don't have to be a game programmer, a programmer at all, or even have a job in computing. You don't even have to use computers, ever. (Well, OK, you do have to use computers, or you wouldn't be here.)

I want you to write down a list of 5 tools you use on a regular basis, to get stuff done. Pick your favorites; it doesn't have to be scientific, just the 5 you can think of quickly that make the biggest difference in your daily life. If you're a programmer, list your IDE, editors, browser, whatever. If you're an artist, list your art software, tablets, pen and paper brands, and so on. If you sit around on your fat ass all day surfing the web and eating junk food, list stuff like "plastic bags," "the Coca-Cola corporation," or "fork."

Got your top 5? Good. Now find out who made each of them. Send the creator a letter of thanks, and make it sincere. If appropriate, send a small cash donation, anything you can afford.


No, this is not a subtle plot - I did not litter the above paragraphs with subliminal messages in an attempt to harvest cheques and praise from you. (Go ahead, read it again - you won't find any such devious devices. I hid them too well! They don't exist.)

I haven't lost my grip on reality, either - fact is, I never had one.

My motivation for this is simple: tools are underappreciated. That may sound weird in a day and age where editor, compiler, and 3D modelling package are defended in epic flamewars that make religion jealous. Who hasn't been singed by the roiling inferno of Emacs vs. vi, Visual C++ vs. GCC, Max vs. Maya? Linux vs. Windows? OpenGL vs. DirectX? Puppies vs. kittens?

Sure, plenty of people understand very well that tools are important, and how much effect they can have. I've dribbled quite a few grandiose tomes about how important tools and efficient workflow can be. But there's still a problem.


I love Fred Brooks. I really do. If you haven't read The Mythical Man-Month, go buy it and read it like ten times, rightfrickennow. It's genius, it's profound insight, and it's crammed full of invaluable truths about how our business works. And it's three decades old. You can't buy timeless power like that. (Well, actually, you can - try Amazon or your local Borders.)

Good ol' Fred suggested an interesting formation for software development teams, similar to the way surgeons do their thing. One of the central members of the Surgical Team is the tool creator. Brooks correctly and emphatically points out just how valuable that guy really is, and makes it clear that to underestimate that role is suicidally foolish.

And yet how many times have we all looked over the classifieds, seen a nice-looking job with great pay and benefits for which we are totally qualified, and then shrugged it aside because "it's just tools"? Shame on us. Shame on us stupid, stupid fools. I've done it, and I'm pretty sure you probably have, too. Heck, if you're like me, you do it a lot - and rarely if ever realize it.


Here's a quick thought experiment for you: think about some huge, awesome, ground-breaking, revolutionary project that you'd be interested in working on. The Pneumatic Midget Cannon, Doom IV, a cure for cancer, holographic interactive porn, whatever. Now, suppose you're given your dream job working on this project: you set your own hours, your own responsibilities, and even your own salary. You can have one of two main roles: working on the project directly, or creating tools for others to use as they do the project. The positions are identical in every possible way, except for the actual work you do.

Show of hands: who wants to make tools?


Put your hand back down, you filthy liar. Everybody wants to be on the front line, doing the sexy job and raking in the TV interviews and book deals. Can you name a single person involved in the Manhattan Project, without looking it up? But I bet you know the president who dropped the bomb. Can you name three people who have worked on Visual Studio? (Microsoft employees need not apply.) But you probably know people who use it... like, say, John Carmack, or Tim Sweeney.

Ever heard of Alexander the Great? Sure you have. Ever heard of Iolaos the Ironmaster? Didn't think so - he was "merely" the guy responsible for creating most of the legendary weapons and armor used by Alexander's mighty armies.

And yet you've never heard of him.


OK, fine... the fact that I totally made the guy up sort of skews the numbers. But it's the principle that counts; I had to make him up because nobody knows who actually did it. Being the guy that makes the tools isn't glamorous or attractive. It's just boring. If you want the glory, you've got to be the guy who goes out and uses the tools.

Some people might argue that that is a natural thing; after all, a sword isn't really worth much unless you cut and stab stuff with it. Compilers can't write code by themselves. Little balls wrapped in leather don't run around autonomously on fake grass fields. People have to use the tools. So who really cares if the tool creators don't get any love?


I say, though, that this situation is a travesty, a tragedy of epic proportions, a horror and shame and all-around detriment to all of civilisation. Well, maybe not all that, but it definitely sucks.

Sure, we have to use the tools in order to do anything meaningful, so tools on their own are nothing. But what is a great man with no tools? I submit to you that he's actually an impotent shmuck who will get fat and die of a myocardial infarction on his couch from eating too much junk food. Or maybe get stabbed by the Other Guy and breathe his last in a decidedly anticlimactic pool of innards. That sort of depends on what exactly he was going to do. But either way, he's probably not going to get much done.



Magnificent tools with no users are, at worst, still pretty good tools. Magnificent users with no tools at all aren't worth jack.

Truman would be nice and obscured by history if he hadn't cooked a couple of cities clean off the map. Had he never gotten ahold of computers and programming tools, John Carmack would have been... well, not nearly as famous most likely. If good ol' Alex didn't have his sword, he wouldn't have done that whole Gordian Knot thing, and we probably never would have heard about him.


There's a moral to all this. The next time you start up some project, don't fall victim to the temptation to do the tools later. I routinely see projects start up, and people are all excited and enthusiastic, and really keen on getting stuff done. They figure it's worth the mild inconvenience of editing things by hand and gimmicking around for a while until the "tools" are all done. After all, maybe the data formats will change, or my ideas will go some other direction, or... or... or... whatever. Write in your own excuses.

The important thing is, those excuses are evil lies. Eventually, you will reach a point where not having tools is actually costing you more time, effort, and frustration than the expense of doing all the "boring" job of creating the tools. I think most people already realize that; the problem is, most people wait too long to do the tools. By the time you notice that you need tools, it's already too late.


Suppose you're out on the battlefield. Maybe you're saving the earth from evil alien demons, or whatever. You brazenly swaggered out onto the field unarmed, certain that as soon as you knew the level of danger, you could run back, order a suitable weapon, wait for it to be manufactured, learn how to use it, then get back out to those evil demons in time to load some hot lead in their maniacal asses.

Sounds really stupid, doesn't it? And yet, in software development - especially games - we do this to ourselves all the time. We start with just a "quick and dirty prototype" and then refine it into a "pre-1.0 version" and so on. Next thing we know, it's legacy code, three generations of maintenance programmers have poked at it, and the users still create data files by hand in Notepad.

If you realize you're hurting for lack of a tool, you've already wasted invaluable and irrecoverable time. It's too late; the demons are already using your bones as toothpicks. And, honestly, when can we not see the need for tools in advance? The alien demon invasion scenario is obvious enough, sure, but what about WidgetMeister 2.0? Is it possible that we might actually not be able to reliably predict what tools we'll need?

Well, you tell me. Personally, I can't think of a single case where the purpose of the end product is known, but we can't figure out what tools will be needed to get the job done. Any good gamer or military buff can tell you that, to repulse an alien demon invasion, you need some Big Fucking Guns. And any good programmer can tell you what tools you'll need, long in advance.


Here's a scenario: excited with the plans for New Project Foo, the team sits around one afternoon with coffee and donuts, brainstorming and talking. Someone mentions the Quux Data that will be so essential to Foo's operation. Developer One says something like, "Yeah! We can do Such and Such and The Other and it'll all be great! And if we use XML or Other Buzzword of Choice we can reap unfathomable benefits in terms of Obscure Business-speak Bullshit Phrase!" And in pops Developer Two with, "Yep... yep... we could even create a really cool GUI drag-and-drop editor that magically creates Quux files and pets kittens, all in an environmentally friendly way!"

And everyone agrees that the editor would be Really Awesome, and We Should Definitely Totally Do It, and such. Then the resident cynic (that's usually me) says something like, "Well, we have to get the data format nailed down first. We should probably do a prototype, too." Somewhere along the line (it may take a few months), everybody agrees that there's enough work to do, and the Quux Editor will get pushed back to Version 2.0. Hey, then we can sell it as a New Feature - value added to justify the exorbitant upgrade price. Perfect plan, right? Quux files can be edited easily in WhateverApp... it's a little finicky and not as user-friendly as it could be, but who cares?


Sound familiar? I've seen this one play out numerous times, and always with the same results: sometime down the line, long after it is too late, we all wake up and realize that some dark hellspawn is cleaning its teeth with our former shin bones. Oops.


So, if you're the kind of person who doesn't read between the lines very well, or has to have things spelled out in explicit sound bytes (or if you just skipped all my palaver to read the last paragraph and get the punch line), here's the point of my little musings:

Build your tools first. Don't let the Legions of Evil pick flesh out of their molars with the shreds of what used to be your leg.
Next Entry Financial Rape
0 likes 16 comments

Comments

jbadams
Well said! You've inspired me to get an earlier start on the resource editor I was planning to develop later in my project.
August 14, 2006 03:36 AM
OrangyTang
Well said indeed. [grin]

This comes at a good point for me, as I'm trying to get the enthusiasm to rework my animation tool (which has got bigger and more complicated, and the custom GUI stuff I was using just isn't good enough any more and needs to be re-written with a 'proper' GUI library). Recently I spend a month rejigging the same tool to get rid of some of the awkward UI stuff, and with only some minor changes it became much easier to use.

I've been thinking about generalising some of my personal tools and releasing them for others to use, you might just have convinced me that this is a good idea. :)
August 14, 2006 07:48 AM
bytecoder
You've forgotten the case in which the project you want to work on is a tool [grin]

edit:
By the way, you need to stop making such long posts and start, you know, working! Hehe. I'm starting to think this is your way of procrastinating.
August 14, 2006 10:13 AM
bytecoder
Oh, and one more thing: kittens own you.
August 14, 2006 10:17 AM
greggman
awesome post.

Just to keep the discussion going. It probably goes without saying but there is such a thing as bad tools. Tools that hurt a project rather than help it. Tools for which it would have been better if you'd use something off the shelf. Something debugged. Etc.

Would you write your own image editing program? I wouldn't. Photoshop may cost $599 but it's worth every penny if you are editing images on a regular basis and no team can afford to reproduce all it's features. That might seem obvious but I know plenty of teams re-inventing the wheel in other areas from 3d editors to asset management system
August 14, 2006 10:22 AM
ApochPiQ
Absolutely; the ills of bad tools cannot be understated. And, chances are, people who make their living building some particular type of tool will probably do a better job of it than you or me, assuming that we work on something else.


bytecoder - I know it looks epic, but these are actually fairly trivial distractions. I haven't been scientific about it, but at a guess my average time for one of these blitherings is between 25 and 30 minutes. Joe watches a sitcom, I pontificate; no serious loss of productivity involved [smile]
August 14, 2006 10:32 AM
Programmer16
Call me a liar will you, I happen love working on tools [razz].

Very nicely written and entirely true. Good work!
August 14, 2006 11:02 AM
Jotaf
Just to be a dissident voice, I could immediately remember the name of the head of the Manhattan project (Robert Oppenheimer -- is that the correct spelling?) but not the name of the president. Ah well. I fully agree with the rest though.

But concerning your previous journal entry, I honnestly expected some more meat after part 1 :P I mean, it's all well and good that we need to write highly parallelized code but I still don't see how to take it out of the theoretical side onto the practical applications. Maybe I'm being narrow-minded but you can't just shun imperative languages like that.
August 14, 2006 01:10 PM
Daerax
For once we disagree at least with regards to one man teams. I dont think there is such a large problem. I see more people wasting time on tools than the game they wish to make. I have done this. A tool is anything that will be an aid to whatever project you may be working on (even if it istelf a tool). There are many font, sprite, world editors while not so many games. as a counterpoint.

An engine is as well a tool if you wish to make a game. One need not look further than that to find examples of people who wish to make a game wasting time on tools. When you are one person, you must divide your time most carefully and if you have a modicum of sense will know when the right time to develop a tool - when it simply no longer does to do it in the way in which you were. A sort of maximal intersection of time spent and productivity gained.

p.s.
That I can think of: robert Oppenheimer led the project, Feynmann, von Neumann, Enrico Fermi, niels bohr, Hans Bethe. The blacksmiths for Alexander were numerous and likely well heralded within their towns and or cities. That literacy was not widespread makes it unlikely that their involvement would be documented and so it comes as no suprise that their names have been lost to history. Carmack and sweeny get alot more media coverage and so are more likely to be known than your average microsoft IDE and compiler writer. Like the blacksmiths of old, there are simply too many of them and their importance to indirect to attract attention from those who direct attention.
August 14, 2006 03:17 PM
ApochPiQ
That's a very good point. I have a sort of intuitive sense that there's a good response, but it isn't coming to mind yet [smile]

Oh, and phooey on you for picking apart my horrible blacksmith analogy [grin] (You are of course totally correct. In fact most of my examples like that are quite skewed and have much more sensible explanations than the ones I've tried to give them. But the overall point is what matters. Poetic license, and all that rubbish.)


Quote:Original post by Jotaf
But concerning your previous journal entry, I honnestly expected some more meat after part 1 :P I mean, it's all well and good that we need to write highly parallelized code but I still don't see how to take it out of the theoretical side onto the practical applications. Maybe I'm being narrow-minded but you can't just shun imperative languages like that.


I can't? Why not?

If you're interested in seeing what I was driving at in that post, I'd highly encourage you to check out some functional languages. L# is a decent way to get your feet wet if you already have a VS product installed. Really most of what I said only makes sense if you have that background to draw on, so I can understand if it seems pretty extreme and kooky for those who haven't used a functional language deeply.

Anyways... you've got me curious now. Prove me wrong [smile]
August 14, 2006 08:50 PM
bytecoder
I'm not so sure about going completely functional, either. It seems that what's important isn't necessarily being functional, but rather being able to determine what is referentially transparent and what isn't. This can easily be accomplished in a non-functional language by just letting the programmer tag functions as transparent or not. Even if you do that, though, I don't really see it as a good idea. Conceptually, this allows the compiler to automatically thread the application, but it seems that doing it this way would thread almost everything, whether or not it needs to be. Let's take an example:


map(some_transparent_function, array)


Let's say array isn't too big, only a few items in size, and some_transparent_function is obviously referentially transparent. Ok, now since we're a compiler, and since the user has been a nice little programmer and tagged all this for us, we can split off each function call into separate threads and sync up when they're all done. Now I'm not expert on how threads work, but if you do this for every piece of code like this, won't the system be swamped with relatively low-work threads? It seems to me that there are only certain cases when actually threading something like this would be worth it, and a compiler certainly can't figure them out automatically. It seems to me that being able to tell the compiler when to thread something like this and when not to--and heck, since you'd have to do this for the map function anyway, you don't even need to declare functions as transparent anymore. The easiest way to achieve this is to just have separate functions that are multithreaded, e.g.:


map_threaded(some_function, array)


Now, as someone who obviously has more experience with threads than I do, is this an actual concern or are my assumptions incorrect?

edit: I'm too used to posting code snippets in emails. I'm gonna have to re-remember to use code tags.
August 15, 2006 11:35 AM
ApochPiQ
Yes, it's a very real issue, but not without solutions. For instance, Erlang defeats this by implementing its own threading model. The problem is, that's largely in conflict with increased processor-level parallelization, where our goal is to have the hardware (not the software) be able to make more decisions about what runs when.

There's more to it than that, in any case. By the time you actually fix the problem, you're basically coding close to a functional style, without any support from the language itself, and without additional benefits. For instance, your very simple example is more or less meaningless in those imperative languages which have no first-class support for higher-order functions - notably C, C++, and Java. But even closely integrated tagging initiatives like OpenMP are unwieldy, don't fully solve the issues, and don't do anything to ease the difficulty of writing massively parallelizable code.

Functional style tends to produce algorithms that are inherently more conducive to parallelization. Imperative style, by contrast, tends to create serial algorithms, because of its heavy reliance on temporal coupling and shared state.


Of course, pure functional vs. mainly-functional-with-some-hacks-so-we-don't-have-to-use-monads is still open for debate. Personally I favor the latter, for two reasons: it tends to be more accessible, and it "thinks" more in line with the way OSes and hardware are likely to be designed for the foreseeable future. So until we see Lisp Machines take over the desktop computing space, truly pure-functional languages are probably not real likely to be the answer.

But we can still do a hell of a lot better job at creating a functional/imperative hybrid than writing botched layers on top of C++.
August 15, 2006 11:59 AM
Daerax
Quote:Original post by ApochPiQ
Oh, and phooey on you for picking apart my horrible blacksmith analogy [grin] (You are of course totally correct. In fact most of my examples like that are quite skewed and have much more sensible explanations than the ones I've tried to give them. But the overall point is what matters. Poetic license, and all that rubbish.)


Yeah i know, i was just being difficult. [razz] muhahaha im over here!
August 15, 2006 02:31 PM
Daerax
As i suspected. type to go hunt down my reply.
August 15, 2006 02:31 PM
bytecoder
Quote:Original post by ApochPiQ
Yes, it's a very real issue, but not without solutions. For instance, Erlang defeats this by implementing its own threading model. The problem is, that's largely in conflict with increased processor-level parallelization, where our goal is to have the hardware (not the software) be able to make more decisions about what runs when.

Isn't erlang mainly designed for massively parallel computers anyway? I can definitely see how this limit will go away once we get to that stage for personal computing, but even then I'm wary of the extra effort required for state even in a language like OCaml.

Quote:
There's more to it than that, in any case. By the time you actually fix the problem, you're basically coding close to a functional style, without any support from the language itself, and without additional benefits.

What exactly is "support from the language" and what benefits do you think would be missing?

Quote:
For instance, your very simple example is more or less meaningless in those imperative languages which have no first-class support for higher-order functions - notably C, C++, and Java. But even closely integrated tagging initiatives like OpenMP are unwieldy, don't fully solve the issues, and don't do anything to ease the difficulty of writing massively parallelizable code.

Makes it look pretty easy to me, so long as you stay away from state.

Quote:
Functional style tends to produce algorithms that are inherently more conducive to parallelization. Imperative style, by contrast, tends to create serial algorithms, because of its heavy reliance on temporal coupling and shared state.

That sounds like something that could be fixed given a language with proper MP support and a community encouraging it (see: goto in basic vs goto in C++).

Quote:
But we can still do a hell of a lot better job at creating a functional/imperative hybrid than writing botched layers on top of C++.

Correct. Botched layers on python here we come ;-)
August 16, 2006 04:57 PM
Will F
Good post. You inspired me to send an email to a couple authors thanking them for writing excellent books. Admittedly the real purpose of the emails was to send them some fixes to add to their errata files for the books, but if I hadn't read this I probably wouldn't have also thanked them.
August 19, 2006 10:30 PM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Profile
Author
Advertisement
Advertisement