Wake Up and Smell the Metrics! A Rant on Metrics-Driven Development in Online Games

Larry Mellon (Virtual Worlds Consultant), Darius Kazemi (President, Orbus Gameworks)

If you don’t know what metrics are, then you should refer to a talk last year by Darius at the Online Games Developer Conference. His slides contain copious notes and offer a great in-depth look into what metrics are and how they’re useful to game development. If you don’t want to take the time to read the slides, the short and skinny version is that metrics are data that you collect, collate and correlate from the game world. How many people are using this item? How many players know this player? How many players that know this player share this item with that player? As you can see, the queries can quickly become complicated and many levels deep into data pulled from a game. How do you know what data to pull? How do you use the data? What effect does this knowledge have on the development of your game?

These are all questions that metrics evangelists Darius and Larry are out to answer for people. Many companies these days that are using metrics are either using them incorrectly or poorly by not using the data to corroborate facts or simply not investing enough developer dollars into the proper tools needed to collect the data that would be useful to them. This is mainly because a lot of studio executives do not understand the need for a metrics system. All the more frustrating is the fact that metrics are being used to great effect in all other industries, including TV, cell phones, casinos… in fact back in 2000, John Boushy, Harrah’s CIO stated “[implementing metrics tracking] is one of the best investments that we have ever made as a corporation and will prove to forge key new business strategies and opportunities in the future.” What’s even more flabbergasting is that by their very nature online games are built from databases – the infrastructure is already there! Companies like Harrah’s have to spend extra millions of dollars to set up the infrastructure their metrics system relies on, whereas games merely need to build on top of existing databases.

Seeing that Darius and Larry have released the slides to this presentation, along with the notes they basically read word-for-word, I’m going to stop here and let you get more from the slides. Larry includes a number of examples with his work on The Sims Online that help drive home the usefulness of a proper metrics system, but one example I don’t believe was included was from Darius when he was working on Dungeons & Dragons Online. He came across a group of designer arguing over this one item in the game that was being abused and throwing things out of balance, and they were trying to decide whether to just cut it or keep it but redesign it so they don’t totally piss off the players who own it. Darius went back to his desk, pulled up that item’s usage in the game along with the number of players who own it and found the numbers to be absurdly low. He came back to the group and told them and the lead designer said “argument over” and they cut the item out.

Coverage by Drew Sikora

Page 5