• Create Account

$50 ### Categories (See All) Like 16Likes Dislike # Why You Shouldn't Be Making an MMO By Mike Lewis | Published Jun 05 2013 07:45 PM in GameDev.net Soapbox Peer Reviewed by (Dragonsoulj, Prinz Eugn, Josh Petrie) mmo game design project size Author's Note: This is an adaptation of a popular but highly controversial journal entry I wrote here almost two years ago. This is solely my own opinion and is in no way intended to be representative of my employer. Enjoy it (or hate it!) for what I meant it to be - a sarcastic backlash against a common aspiration of game development newcomers. ## The MMO Gold Rush It's nothing new, this trend, but it also doesn't seem to be going away as quickly as most fads; perhaps we can accuse World of Warcraft for making this such a huge deal, or maybe it all goes back to Ultima Online and Everquest. We may never know for sure. One thing is definitely clear, though, and that's that a significant contingent of people show up here at GDNet wanting to make their own MMO games. There have been many words bandied about in the past over this subject, and plenty of advice warning newbies not to get their hopes up when it comes to shipping a true MMO. Sure, not everyone is committed to the RPG genre; some people want to make an FPS, or maybe even an RTS. Others might have even more creative and unique ideas for how their game should play. But the real issue here is not the gameplay itself: it's the MMO part. More than that, it isn't even the "multiplayer" or "online" that is really an issue - it's the "massive." ## The Real Issue MMOs are expected to host dozens of servers running thousands of players apiece. Successful games may be played by over a million people; some notable ones are played by far more than that. Even a low-grade MMO serves a few hundred thousand players. With modern technology and toolkits, it's entirely in reach of a lone wolf or a small team to create an incredible game that can host even a few hundred players. However, as wiser men than myself have observed, "a difference in quantity can often mean a difference in kind." Consider the original airplane, as developed by the Wright brothers over a century ago: a simple wood and canvas structure with a (relatively) lightweight and effective engine, with some simple control mechanisms. Now consider a jet liner. Both are flying machines; but in order to comfortably handle two hundred passengers, the airplane had to do a lot of changing over the years. Even a fighter jet is a different beast. Just changing one quantity - speed - leads to qualitative differences in many ways. A fighter jet has different engines, different control surfaces, different kinds of fuel, different training to fly, and on and on. Go one step further: a fighter jet that's invisible to radar again has many, many differences (ranging from shape to materials to operating principles) than a traditional jet. Many people have set eyes on the available game-development technologies of today, and noticed that they can put a hundred players in a world and turn them loose. The assumption that you can just twiddle some knobs and tweak some sliders to get an MMO is deeply flawed. As a personal anecdote, I once set out to write a realtime ray-tracing engine. I had experience with a popular graphics package called POV-Ray, back when it was still a relatively fresh spinoff from DKBTrace. Since POV-Ray represented an early incarnation of a model of what we would now call "open source", I had access to the code; my plan was to "optimize" POV-Ray and turn it into a realtime system. What I failed to understand was that the difference in speed I wanted represented a tremendous change in how the code had to be written. I managed to eke out about a 5% speed gain over the original POV-Ray before giving up and writing a fresh engine from scratch. (For the record, I did achieve realtime ray-traced rendering in that project, but it was truly hideous and awful code. I still have it someplace but if I remember correctly I left it in a horribly broken state. It's probably better that way. Overall, though, the experience was incredibly educational.) ## The Challenges It's a far cry from 16 people playing Halo on a home-rigged LAN to the kind of stuff a real MMO must cope with. Even throwing in something like Xbox Live Matchmaking is incredibly resource-intensive. Companies like Bungie have people whose full time jobs involve doing very little but making sure the matchmaking experience runs smoothly. When a major company has to employ people to do nothing but run their multiplayer system, you can bet it's a good sign that Little Old Joe in his garage or basement isn't going to be up to doing something similar. But wait! Even running a matchmaking system for a few hundred thousand players pales in comparison to what an MMO must do. After all, a matchmaker simply needs to join up different players and get them running on a separate system (usually hosted by one of the players themselves, or maybe a dedicated server for PC games). If dedicated servers are involved, the matchmaker is often really just a catalog of what severs are running. This is the kind of thing that a lone wolf coder could easily accomplish on his own; something like Xbox Live matchmaking, however, is getting past the realm of stuff that one guy (or gal) could ever do alone. So if a solo developer can't really hope to build, deploy, and run a large-scale matchmaking service, what about an MMO? Let's take the demands of a typical matchmaking service and chuck in a handful of complications. • Persistent Worlds. Instead of just shuffling through a list of players and handing out matchups, which may last a few rounds at the most, an MMO must store data on every player and every action ever made. Sure, many modern games store statistics like how many shots you fired, where you died most often, and so on; but even that is a huge leap of data warehousing beyond what a typical small team can hope to build and maintain by themselves. • Active Simulation. A matchmaker can often get away with having a player host the game, which is the typical approach used in console games these days. Matchmaking servers don't have to have any code besides the stuff that links people together, making the actual multiplayer portion fairly straightforward by comparison. Again, remember that for up to about a dozen players, off-the-shelf network code is often sufficient to get a decent game experience. It does require some skill to make it happen, but it's at least plausible that a single person (or a small team of people) could get away with building a good low-scale multiplayer game. • Massive Scale. We're not talking about sorting through a few thousand people to create a 10-man match; we're talking about having thousands of people connected to a single server, all constantly doing things. Combine this with persistence and active simulation, and stuff just got a lot more painful. ## An Amusing Magic Trick Here's an educational homework exercise for you. Even better, it has a magic trick twist to it. Pick your favorite MMO, and go look up the game credits, either in-game or on the web or wherever. Count the number of people involved. Don't cherry-pick either - look at programmers, artists, musicians, producers, designers, marketing folks, operations/IT staff, and anyone else you can find who participates in a non-trivial way in making that game happen. (You can ignore janitorial staff for the purposes of this exercise.) Now, don't tell me your number. Keep it secret. Without talking to you in any way - before you even do this exercise - I will know more about your number than you might be comfortable admitting. In fact, I'm so sure about this number, that I'll even make you a bet. I will bet you a beer (or suitable beverage of your choice) that you can't find an MMO with over a million players with a development team of less than 100 people. Here's another magic trick. Go find the lead developers and senior-level programmers, artists, producers, musicians, and other high-ranking folks from your MMO's team. Now go find out what other games those people worked on before the MMO. Do the same thing for the less-senior guys as well; average up how many titles everyone has worked on in their past. I will bet you a second tasty beverage that the average developer working on an MMO has shipped at least 2 games prior to shipping a successful MMO. Many successful MMO titles are even the results of collaborative efforts from dozens of people with prior MMO experience. ## A Case Study in Real MMOs At its peak, the Guild Wars 2 team included something like 300 people - and that doesn't account for those who came and left over the course of the game's development. The codebase is many millions of lines of code (I don't have exact figures, and probably couldn't release them anyways.) There are terabytes of assets - ranging from artwork to music to design documents to configuration files. If that scale alone isn't enough to convince you that MMOs are out of reach for the Five Amigos, then let's talk business. MMOs are fiercely competitive, and a cutthroat business. People are doing all manner of wildly innovative and devious things to make money in the MMO space, and most of them are failing. There are some fascinating studies of how people move between MMOs, and they tend to suggest that many people are constantly on the hunt for the next great big thing - meaning that they'll bail on a mediocre MMO in a heartbeat to go play something cooler. Running an MMO is immensely expensive. Internet hosting and server costs alone can be in the tens of thousands of dollars a month range. Buying all the hardware you need to run the game up-front can be well into the millions. You need a dedicated datacenter for the endeavor, with redundant power, fire safety systems, industrial cooling, and hundreds of miles of both copper cabling and fiber optics. A single network switch capable of running an MMO backbone can cost ten grand by itself. And if you want a global reach, you'd better roll out a datacenter on every major continent, at the very least. I'll gloss over the details like how to go about creating all the assets and content necessary to flesh out a true MMO, and stick with the technical specifics (which is where I spend most of my time anyways). At this level, everything becomes important. The difference between allocating space for a buffer on the stack versus the heap is a crucial design decision. Calling conventions must be selected with great care. Exception handling is nearly verboten, because it has too much runtime overhead and can create unpredictable execution flows in code that must run continuously for years at a time. One-byte memory leaks are show-stopper, can't-ship-the-game-like-this bugs - because that one byte will kill you when it leaks constantly for eight months. Microseconds of sloppily implemented code turn into milliseconds of delay when scaled across thousands of players, which adds up to lag - the number one evil of an MMO. You'd better be a sheer wizard at lock-based concurrency, because the order in which you lock things might spell the difference between a happy server and a worldwide outage. For that matter, you'd better be a wizard at lockless concurrency too, because locks are often too expensive and too exclusive to scale to thousands of active connections. Memorize those atomic operations, folks, and if you still think "volatile" means anything useful, go back to Go and do not collect$200.

You'd better understand how operating system kernels work, because you'll be taxing the living hell out of yours. Your job will bank on you knowing things like pre-emptive multitasking architectures, monolithic versus micro-kernels, how kernel-versus-user-mode code works, and how schedulers are implemented.

Better brush up on your compiler theory, too, because you'll need to know exactly how your code gets converted to machine language. For that matter, you'd better know your assembly language and your computer architecture, too. Two or three clock cycles here and there will add up in a damned impressive hurry. Don't know what a superscalar CPU architecture is? Pipelining? L1, L2, and L3 caching? Don't know how a memory bus is implemented? Don't know the difference between a northbridge and a southbridge? Don't know the role of device drivers and how they work? Don't know where the network stack of your OS of choice lives and under what context it runs? Pack your bags and don't sell the bike shop, 'cuz you aren't cut out to be writing an MMO server.

It takes half a dozen dedicated programmers just to write the servers for Guild Wars 2 - and that doesn't include the people who wrote some of the legacy code we inherited and modified from GW1, or the people working on auxiliary support features like web sites, forums, wikis, and so on. It doesn't even include the network admins who build the hardware and maintain it, or the community managers who take care of ensuring that the experience is enjoyable for all the players.

The guys I work with are, to a soul, incredibly good programmers. I'm still not entirely convinced that I'm cut out to be working with them, or on a project of this magnitude. It's without question an extreme challenge - but one that I'm tackling with gusto, because I love a good challenge. There's more brilliance packed into the room I sit in every day than I've seen in years in the industry, and I've worked with some damn good people in the past.

## The Moral of the Story

My goal here isn't to scare off everyone who doesn't have a pile of money and time at their disposal; my goal is to filter down to the people who are motivated enough to put in that time. I'm not trying to discourage people from pursuing their dreams - I'm just trying to help people self-select their dreams and maybe try leaping at some smaller goals before shooting for the moon.

Individuals interested in accomplishing great stuff need to first prove themselves - and hone their skills - in smaller arenas. Some may not agree, which is fine, but harsh experience and observation of many successful people in the business (as well as many people who failed to go anywhere) has taught me that the only certain way to large-scale success is to master the small first. Yes, maybe some obscure genius did the big thing once without ever learning the basics, but those are exceptions, and rare ones at that - in my mind, it's a disservice to newcomers to any craft to tell them that they can be just like that, because in the majority of cases it simply isn't true.

I'm not arguing that individuals or smaller teams can't make awesome games - there's ample evidence that they can and do. Even multiplayer games are not out of reach. My point centers entirely around the "massive" label.

By all means - go make games. Make fun, awesome, compelling, memorable, and successful games.

Just be mindful of the fact that success is equal parts intensely hard work, and knowing how to pick your battles.

GameDev.net Soapbox logo design by Mark "Prinz Eugn" Simpson

Mike Lewis has done time as a senior server programmer, security coordinator, and lead gameplay programmer at ArenaNet, makers of Guild Wars 2 - a decidedly massive online game.

Kind of an outdated article that I'd sign in a heartbeat if it was 2004.

But with todays advance in technology some things are possible on a smaller scale.

The critical changes are:

- Hosting

as this is easier now with the power of the cloud (Btw. aren't you hosting GW2 on Amazon EC2 servers? I thought I read something in an developer MMA once on reddit)

- Engines

UE3 and Unity can handle a lot of stuff

- Bandwidth usage

the smaller the better, but you have a lot more freedom now

- High level languages

quicker development times at the cost of performance (that could spawn a new discussion if that's really a good argument for MMO development but I think that C++ strengths are also its downfall in complex game environments)

The first 2 points are complete game changers for a small company. They weren't even possible 3 years ago.

I agree that you can't make something like WoW with a small team. The sheer amount of content you have to design is just not manageable. From a technical standpoint I do think it's possible.

The whole point of the article is aimed at "massive" scale.

Cloud hosting does not scale for MMOs. You need dedicated hardware, and at the prices cloud providers charge for the kind of horsepower required, it's cheaper to run a colocated datacenter. GW2 is not hosted on EC2, for the record.

UE3 and Unity are nowhere near powerful enough to scale to more than a hundred or so players in a map, and they need some serious TLC to get that far in the first place, especially Unity.

Bandwidth is still a hard problem if you have more than a few dozen clients. It's just the nature of physics at that point.

Language choice is virtually irrelevant; you still need the technical power to do the things I talked about, and you definitely need good programmers, so that's really not much of a thing.

And as I tried to emphasize repeatedly: I'm not talking about making games with a few dozen concurrent players or even something like a 2D graphical MUD. I'm talking about AAA-scale MMOs. Small scale is totally in reach, and mid-scale is a lot less scary than it was even five years ago; but massive scale is a different story altogether.

I tend to agree with you on this topic a lot. And as it turns out, you were right on this back when you wrote your first material.

The overall summary is that the MMOs require too big of an investment and the support is too costly. The market is tight mostly due to the already established (for years and years) leaders and getting your investment back, let alone profit, is going to be a huge and hard thing. Even big and proven studios/publishers in this field (for example NCSoft and I love most of their production) have fallen short on some of their AAA titles.

Thanks for the good read, the article was very enjoyable.

Can you elaborate why cloud hosting doesn't scale?

Great read though I have a small problem; even though this article is perfectly valid, there are going to be many budding massive multi-player online developers who have convinced themselves that this does not apply to them because they are starting off "small" (which is a contradiction).

While thinking about how I'd re-enforce this message, I realised that the only way you'd convince the type of people you are talking about would be to do something like give them a few days work experience in a team working on a real game (a single day wouldn't work as they'd likely still be starry eyed at 8pm). I reckon by day 3 they'd be asking to go home by 5.

Here's the thing though, do we want to crush peoples hopes? Even though the message is valid, I'd rather be part of a community that dreams of making these epic games, than be part of one where negative re-enforcement is used to suppress imagination. I personally started out with delusions of grandeur as well, and learnt without anyone telling me what I was actually capable of over time.

You know, I'd go as far as saying the scope of a game idea tells you how young a game developer is.

My 2 cents? let them dream ;)

Honestly, I'm totally cool with crushing people's dreams.  Why?  Some people literally have no idea what it takes to build even a small, non-networked game.  Ask the average person how long it takes to build a small mobile game, and they'll respond with some ridiculous figure.  My favorite is to ask how much money did it take to build a game and often people respond with $100 -$1000 total.  Given minimum wage... yeah, somebody built that game in that short of a time.

There were scores of students in my CS department's game making club, and every single year we'd get half of the game suggestions to be something about a MMO game.  It was just so ridiculous how little people understood the difficulties in building a high quality game.  But we shouldn't dream crush just to feel like we're smarter than other people.  We always told people that building such games were outright crazy if you wanted to actually finish it and that they probably did not have the requisite experience/knowledge to be even capable of building such a thing.  Afterwards, we usually recommended some classes that would help them get the basics necessary to possibly build portions of the game (hey, take some networking classes, some OS classes, algorithms, compiler construction, etc).

Some people were either too stubborn to change their minds or far too inexperienced to even realize they were inexperienced or lacked the skills necessary to build a large scale MMO game.  Those people usually weren't seen at the club for weeks or months and then we'd see them again toward the end of the year and they would tell us that they tried to build their MMO and in less than 1 or 2 weeks, they realized they couldn't do it and gave up.

You need to have realistic expectations and understand your actual skill level.  Build things that are within or just slightly beyond your capabilities, not things that are massively (no pun intended) beyond them.

Just a quick claryfication : What size game do you think 4 or 6 guys can pull off?

If I read you right it seems that when most people speak of developing an MMO , they may only be realistically hoping for a couple 1000 concurrent players if everything goes awesome for them.

Note: GameDev.net promotes a collaborative environment.

PARTNERS