Jump to content

April 2017 »

232425262728 29
- - - - -

The Poor Man's Gameplay Analytics

playtesting analytics testing user testing
4: Adsense

You don't want to take time away from your awesome game to write boring analytics code. So you either call up some friends, hire playtesters, integrate some 3rd party SDK (ugh), or just do without.

WRONG. You roll your own solution. Here's why.

Why you don't do without

The OODA loop models the way human individuals and groups operate. It goes like this:
  • Observe the situation
  • Orient your observations in the context of goals, past experience, etc.
  • Decide what to do
  • Act on your decision
  • Repeat!
After the initial act of creation, game developers operate under the same principle. You play the game and observe it, orient that data in the context of your goals, decide what to do, open your IDE, and put your plan in action. Rinse and repeat.

The problem is, unless you're a god-like game designer, your observations are fatally flawed, and there's nothing you can do about it. You're so close to the game that your view is wildly different from your players. You're sick of mechanics that should be fun. You breeze through sections that should be brutally difficult. Subtleties are blindingly obvious. Frustrating glitches are comfortable and familiar.

Your only options are to either reincarnate as Tim Schafer, or seek outside perspectives.

Why you don't hire playtesters

Easy: because you're broke.

Why you don't bring in friends

You can't trust yourself in this scenario. Here's how it always plays out:

You run into someone on the street. Maybe you know them, maybe not. Either way, you're about to monopolize their time.

You: Hey, want to playtest this game I'm working on?

Hapless victim: Um sure, I mean, I don't play video games much but

You: Don't worry, it's a procedurally F2P permadeath MOBA, you'll pick it right up.

Hapless victim starts struggling through tutorial. You hover behind them, watching and breathing heavily.

Hapless victim: This game is... cool. Very cool. Whoops! Died again. Haha. Hey I have to go to work/class/other vague commitment, but cool game! Very cool.

When you hand your game to a friend or stranger, you're saying "I think this is worthy of your time and I'd like to show it off to you." But the reality is, it's not (yet). It's incomplete and probably buggy. They try so hard to like it. You try so hard to convince yourself that they like it, but end up with hurt feelings anyway. Any useful data points you salvage from the wreckage are tainted by the fact that you were watching over their shoulder.

To solve these problems, you send a build to your close friend with instructions to ignore the bugs and placeholder assets, then breathlessly await their response. Maybe you get some useful feedback, maybe not. Either way, it's a big favor to ask of a friend, and you only get one new flawed perspective. Also, they're only a first-time player once.

Why you don't use third-party SDKs

Most third-party SDKs work great for F2P games where all you want to see are big-picture things like ARPU, retention, crash logs, etc. By all means, use them to get those numbers.

But to truly observe, you need to be able to watch over the player's shoulder without affecting their gameplay. Anonymously of course, and with their permission. You need to be able to record the whole gameplay session and play it back, and most third-party SDKs won't do that. The best way is to write analytics playback code directly into your game or level editor, because it already has the same code and assets.

One notable exception is GameAnalytics (if you're using Unity), which is free and offers 3D heatmaps right in the Unity editor.

Roll your own

It's the only remaining option. Send your build to friends, randos, whoever. No need to provide instructions, no awkward conversations. It's detached and impersonal, and you get exactly the data you need, because you wrote code to collect it.

What follows is my experience building custom analytics with not much time or money.


It's easier than you think. For each gameplay session, there are three types of data you need to track:
  • Metadata - timestamp, duration, version number
  • Events - just a timestamp and an event name
  • Continuous attributes - things you want to graph continuously over time, like player position and health
I use C#, so I shove this data in a classand serialize it to an XML file using built-in .NET functions.
Having one file per gameplay session makes it easy to move your analytics data around. If the player doesn't have an internet connection, you can retry later, deleting each file after uploading it to avoid duplicates, like this:
string url = "http://your-server.com/" + Path.GetFileName(file);
new WebClient().UploadData(url, "PUT", File.ReadAllBytes(file));
On the server side, I store these files in Amazon S3 for something like 50 cents a month. I don't want to give my S3 credentials to the client, so I have a dead simple Django app that accepts file uploads and uses boto to put them in my S3 bucket. It's hosted for free on Heroku. Here's the entire source code:
from boto.s3.connection import S3Connection
from boto.s3.key import Key
import boto
from django.conf import settings
import mimetypes
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt
def store_in_s3(filename, content):
    conn = S3Connection(settings.S3_ACCESS_KEY_ID, settings.S3_ACCESS_KEY_SECRET)
    bucket = conn.get_bucket(settings.S3_BUCKET)
    k = Key(bucket)
    k.key = filename
    mime = mimetypes.guess_type(filename)[0]
    k.set_metadata('Content-Type', mime)
def upload(request, filename):
    if request.method == 'PUT':
        store_in_s3(filename, request.raw_post_data)
        return HttpResponse(status = 200)
        return HttpResponse(status = 403)
Wait for the sessions to roll in, download them from S3 (I recommend S3 Browser if you're on Windows), and load them into your game.


Posted Image

The UI is all custom, but it really wasn't too bad to write. On the right-hand side, I can view individual play sessions, or select all of them and filter by event type. For example, to see where people most often exit in a fit of rage, I can select all the sessions, then filter on the “Exit” event. The events appear on the timeline at the bottom, where I also graph the average of each continuous attribute (just health for now).

I also visualize player position at the time of each event as a 3D colored dot in the game world. I take the MD5 hash of the event name and normalize the first three bytes to generate the color. I can also play back the recorded sessions at up to 10x speed, and literally watch the position of each player as they go through the game.
The data collected has been invaluable for discovering things that no one will ever tell you in a feedback form. Things like "the second ledge on level 3 is a bit too high". In short, it's worth every second spent.

So. Go forth and measure!

Mirrored on my blog

Feb 20 2014 12:54 AM

Sounds like a great implementation, thanks for sharing! :)


This would probably make a great article if you're interested in submitting it there as well!

Feb 20 2014 07:18 AM

Thanks, I'll do that!

Feb 20 2014 06:29 PM

That's really great!


I wonder -- how much playback is your data able to support? Are you actually able to watch the entire gameplay session in-engine (much like Halo's recent playback feature) or do you just have temporal time/event markers that give you some idea of the flow a player takes?

Feb 20 2014 10:29 PM

I can't playback the whole gameplay session because I don't record the state of every single object, I just record the player's position along with event markers. So I can watch the player's position move through the level in realtime, and I can see when they perform certain actions, but I can't see which way they're facing, or the position of any other object in the game.


This is where it's nice to write your own solution, because while this works great for my game, another game might have different requirements. You can record exactly what you need to see without worrying about shoe-horning it into some third-party service.

Feb 21 2014 10:20 AM
It looks like this post is an indirect playtesters recruitment...
Where do I sign?
Feb 21 2014 01:31 PM

So, Halo gets away with relatively small recording files because they've taken pains to make the simulation deterministic (Lots of RTSes do this too, because otherwise they've have so much state flying around). In short, they record the random number seed at the beginning of the record and then make sure that their random number gathering is deterministic -- then they don't have to record state for all events, just the ones the player is directly in control of.


It may not be something that makes sense for your game now, but the two techniques could be really powerful together. I'm very much convinced of doing this in my next project.

Feb 21 2014 08:44 PM

It looks like this post is an indirect playtesters recruitment...
Where do I sign?


Haha! It wasn't intended as such, but if you want I'll add you to the list. :)

Note: GameDev.net moderates comments.