Get some unsolicited advice from a guy working on a real MMORPG.


It's nothing new, this trend, but it also doesn't seem to be going away as quickly as most fads; perhaps we can accuse World of Warcraft for making this such a huge deal, or maybe it all goes back to Ultima Online and Everquest. We may never know for sure.

One thing is definitely clear, though, and that's that a significant contingent of people show up here at GDNet wanting to make their own MMO games. There have been many words bandied about in the past over this subject, and plenty of advice warning newbies not to get their hopes up when it comes to shipping a true MMO.

Sure, not everyone is committed to the RPG genre; some people want to make an FPS, or maybe even an RTS. Others might have even more creative and unique ideas for how their game should play. But the real issue here is not the gameplay itself: it's the MMO part. More than that, it isn't even the "multiplayer" or "online" that is really an issue - it's the "massive."

MMOs are expected to host dozens of servers running thousands of players apiece. Successful games may be played by over a million people; some notable ones are played by far more than that. Even a low-grade MMO serves a few hundred thousand players.

The Real Issue
This is the root of the problem, here, this "massively" multiplayer business. Because going from just "online" and "multiplayer" all the way up to "massive" is a huge deal. Serving a pair of co-op players is fairly easy; most off-the-shelf engines these days include at least rudimentary networking support, and it doesn't take a huge amount of network kung fu to get a co-op experience running smoothly for most game designs. Half a dozen players isn't a big deal, either, using existing networking solutions.

But it's a far cry from 16 people playing Halo on a home-rigged LAN to the kind of stuff a real MMO must cope with. Even throwing in something like Xbox Live Matchmaking is incredibly resource-intensive. Companies like Bungie have people whose full time jobs involve doing very little but making sure the matchmaking experience runs smoothly. When a major company has to employ people to do nothing but run their multiplayer system, you can bet it's a good sign that Little Old Joe in his garage or basement isn't going to be up to doing something similar.

But wait! Even running a matchmaking system for a few hundred thousand players pales in comparison to what an MMO must do. After all, a matchmaker simply needs to join up different players and get them running on a separate system (usually hosted by one of the players themselves, or maybe a dedicated server for PC games). If dedicated servers are involved, the matchmaker is often really just a catalog of what severs are running. This is the kind of thing that a lone wolf coder could easily accomplish on his own; something like Xbox Live matchmaking, however, is getting past the realm of stuff that one guy (or gal) could ever do alone.

So if a solo developer can't really hope to build, deploy, and run a large-scale matchmaking service, what about an MMO?

Well, let's take the demands of the matchmaking service and chuck in some complications:

• Persistent worlds. You're no longer hosting a game for a few rounds tops; you're storing data on players and their every action in the game. This introduces all manner of data warehousing issues which don't affect non-persistent games. Even semi-persistent shard/instancing systems (say, like that used in Guild Wars) are pretty complex.

• Active role in simulation. A matchmaker can often get away with having a player host the game, which is the typical approach used in console games these days. Matchmaking servers don't have to have any code besides the stuff that links people together, making the actual multiplayer portion fairly straightforward by comparison. Again, remember that for up to about a dozen players, off-the-shelf network code is often sufficient to get a decent game experience. It does require some skill to make it happen, but it's at least plausible that a single person (or a small team of people) could get away with building a good low-scale multiplayer game.

• Massiveness. We're not talking about sorting through a few thousand people to create a 10-man match; we're talking about having thousands of people connected to a single server, all constantly doing things. Combine this with persistence and active simulation, and stuff just got a lot more painful.

Some Magic Tricks, For Your Amusement
Here's an educational homework exercise for you. Even better, it has a magic trick twist to it. Pick your favorite MMO, and go look up the game credits, either in-game or on the web or wherever. Count the number of people involved.

Now, don't tell me your number. Keep it secret.

Without talking to you in any way - hell, before you even do this exercise - I will know more about your number than you might be comfortable admitting.

In fact, I'm so sure about this number, that I'll even make you a bet.

I will bet you a beer (or suitable beverage of your choice) that you can't find an MMO with over a million players with a development team of less than 100 people.

Here's another magic trick. Go find the lead developers and senior-level programmers, artists, producers, musicians, and other high-ranking folks from your MMO's team. Now go find out what other games those people worked on before the MMO. Do the same thing for the less-senior guys as well; average up how many titles everyone has worked on in their past.

I will bet you a second tasty beverage that the average developer working on an MMO has shipped at least 2 games prior to shipping a successful MMO. Many successful MMO titles are even the results of collaborative efforts from dozens of people with prior MMO experience.




The Uncomfortable Questions That Must Be Asked
If a typical heavy-hitter MMO needs more than 100 people to make it, how is it in any way feasible for one person to do it alone? What about 5 people? 20? At what point do we "break even" and hit a level of team size that might possibly tackle one of these suckers?

In a similar vein, if the average developer is already experienced with at least a game or two, and maybe even a prior MMO, what makes you think that a total newcomer to game development has any shot at accomplishing such a gargantuan task?

And these are just the obvious, logical questions. I haven't gotten to the heavy stuff yet.


A Case Study in Real MMOs
Let's look at my current employer, ArenaNet, producer of Guild Wars and the upcoming Guild Wars 2.

The GW2 juggernaut employs over 250 people. The GW2 codebase is several million lines of code (I don't have an exact count yet; I might get around to running a full scan of the code and put together a snapshot of how big the codebase is later on). There are hundreds of gigabytes of assets, ranging from artwork to music to design documents to configuration files.

If that scale alone isn't enough to convince you that MMOs are out of reach for the Five Amigos, then let's talk business.

MMOs are fiercely competitive, and a cutthroat business. People are doing all manner of wildly innovative and devious things to make money in the MMO space, and most of them are failing. There are some fascinating studies of how people move between MMOs, and they tend to suggest that many people are constantly on the hunt for the next great big thing - meaning that they'll bail on a mediocre MMO in a heartbeat to go play something cooler.

Running an MMO is immensely expensive. Internet hosting and server costs alone can be in the tens of thousands of dollars a month range. Buying all the hardware you need to run the game up-front can be well into the millions. You need a dedicated datacenter for the endeavor, with redundant power, fire safety systems, industrial cooling, and hundreds of miles of both copper cabling and fiber optics. A single network switch capable of running an MMO backbone can cost ten grand by itself. And if you want a global reach, you'd better roll out a datacenter on every major continent, at the very least. Three or four per continent is more like it.

What about technology?

Graphics are amazingly critical. If you don't have a great looking game, don't expect to attract too many players. The MMO space more than any other genre is dominated by players who are into aesthetics and first impressions. A great game with bad graphics might still get some hardcore fans, but it'll never compete with a great game with great graphics.

If you're just a mediocre game, though, the prettiest pictures in the world won't save you, so don't skimp on other stuff. Physics is becoming increasingly relevant. Audio must be immersive and high-quality. UI must be seamless, well-polished, and intuitive.

And we haven't gotten to the networking side of things yet.

The client side is pretty easy; it just has to connect to a server and spam some packets back and forth. But the server itself is a place of truly dark voodoo.

I've spent a grand total of two days with my head buried in the guts of the Guild Wars 2 server code, and I can say this much: there's more going on here than any one brain can keep track of. The GW legacy includes some truly brilliant, genius-level programmers, and even they didn't pretend to know all about the entire system. Sure, most of the veterans on the team have a good high-level picture of how it all fits together, but the details? Forget it.

At this level, everything becomes important.

The difference between allocating space for a buffer on the stack versus the heap is a crucial design decision. Calling conventions must be selected with great care. Exception handling is nearly verboten, because it has too much runtime overhead and can create unpredictable execution flows in code that must run continuously for years at a time. One-byte memory leaks are show-stopper, can't-ship-the-game-like-this bugs - because that one byte will kill you when it leaks constantly for eight months. Microseconds of sloppily implemented code turn into milliseconds of delay when scaled across thousands of players, which adds up to lag - the number one evil of an MMO.

You'd better be a sheer wizard at lock-based concurrency, because the order in which you lock things might spell the difference between a happy server and a worldwide outage. For that matter, you'd better be a wizard at lockless concurrency too, because locks are often too expensive and too exclusive to scale to thousands of active connections. Memorize those atomic operations, folks, and if you still think "volatile" means anything useful, go back to Go and do not collect $200.

You'd better understand how operating system kernels work, because you'll be taxing the living hell out of yours. Your job will bank on you knowing things like pre-emptive multitasking architectures, monolithic versus micro-kernels, how kernel-versus-user-mode code works, and how schedulers are implemented.

Better brush up on your compiler theory, too, because you'll need to know exactly how your code gets converted to machine language. For that matter, you'd better know your assembly language and your computer architecture, too. Two or three clock cycles here and there will add up in a damned impressive hurry. Don't know what a superscalar CPU architecture is? Pipelining? L1, L2, and L3 caching? Don't know how a memory bus is implemented? Don't know the difference between a northbridge and a southbridge? Don't know the role of device drivers and how they work? Don't know where the network stack of your OS of choice lives and under what context it runs? Pack your bags and don't sell the bike shop, 'cuz you aren't cut out to be writing an MMO server.

It takes half a dozen dedicated programmers just to write the servers for Guild Wars 2 - and that doesn't include the people who wrote some of the legacy code we inherited and modified from GW1, or the people working on auxiliary support features like web sites, forums, wikis, and so on. It doesn't even include the network admins who build the hardware and maintain it, or the community managers who take care of ensuring that the experience is enjoyable for all the players.

The guys I work with are, to a man, incredibly good programmers. I'm still not entirely convinced that I'm cut out to be working with them, or on a project of this magnitude. It's without question an extreme challenge - but one that I'm tackling with gusto, because I love a good challenge. There's more brilliance packed into the room I sit in every day than I've seen in years in the industry, and I've worked with some damn good people in the past.


The Last Straw
I haven't even started to touch on content production, or quality assurance, or game design in general, or balancing, or audio, or art creation, or anything relating to administration and running a business. Those things are all major tasks, outside the already-mammoth enterprise of programming. I've focused on code because that's what I do, but it's by no means the largest, most difficult, or most important aspect of making an MMO.

So, hopefully, that gives you some idea of what you're up against when you waltz into For Beginners and say, "hey, me and my three friends from high school want to make the next World of Warcraft."

Sure, with enough time, dedication, and discipline, you might wind up on a team that does produce the next smash-hit MMO. But you'll probably make up around one percent of the workforce on that team. And you'd better be fucking good - and I mean that.

Please, in all seriousness, start with Pong. Make a good Pong game. Make a killer Pong game. But scheming to produce an MMO from your basement is a surefire recipe for harsh lessons in reality, and for failure. It would be a damnable shame to waste your enthusiasm and talent on a doomed foregone conclusion of a project.

Instead, go make something small. Make something cool. Make something fun. Then, perhaps, you'll see that the important thing isn't massive numbers of players, or huge server farms, or millions of dollars. The important thing is that you do something you really enjoy with your life and your time.


And if you're sick in the head enough to enjoy something like working on an MMO, come apply to ArenaNet. I hear we're still looking for a few good senior programmers.


POSTSCRIPT
Check out my responses to the comments.