You can read other articles in this series here:
Part 1: Design & Prototyping
Part 2: Building the Core
Part 3: Balancing & Polishing
You can find out more about Mirthwerx and our projects at our website.
We had a game! Hurray! It worked, it played well, it has a beginning, middle, and ending. We were ready to get the sucker out the door. But before release, we had to go through the final stage of development: testing. Wow, what an eye opener!
No one Knows How to Play Your Game, and They Don't Care to Learn!
I submit I may be the strangest person alive.
I grew up in the 80's and early 90's. This is where many of my early gaming habits formed. Back then when I bought a new computer game, such as Civilization, Ultima, or Wing Commander, I would sit down and read the entire manual cover to cover before attempting to play the game. And this was when manuals were works of art: the civilization manual was over 100 pages and filled with fascinating sidebar historical facts. I continued this practice, though I've stopped now that the manuals are nothing more than epilepsy warnings and telling me where the square button is on my controller. (Fortunately board games still have amazing manuals, so I can enjoy those )
The good old days when games were hard and manuals were thicker than your arm!
Apparently no one else read manuals so the gaming industry moved away from them altogether. Players want to play, not learn how to play. I sort of knew this, I read about it in game design books, but I didn't have the experiential knowledge of it. It quickly came.
At our pre-release party (unfortunately 3 months before the actual release, but who's counting) we had several iOS devices with a 4 level demo version of Catch the Monkey installed on them for people to play. We watched people pick up the game and play it for the first time. We learned two things: people enjoyed playing with the monkeys, but they had no clue how to do it. So while they had fun, they were frustrated not knowing how to use the various tools.
It seems obvious as I write this, but we discovered our need for tutorials built into the game. All I can say is that when you work on something closely for 2 years you lose track of what is "intuitive" and what isn't. Observe strangers and reality will come crashing in. So, we used our character dialog system to retrofit in a tutorial system.
First Iteration of Tutorials. Nobody reads em.
Weeks before release, we tested with a focus group of teenage girls (our demographic) to see how they enjoyed the game. I squirmed in my seat as I watched them tap "next, next, next" and completely bypass the tutorial to get on to the game. Once there, they didn't know how to use certain features, started losing, and became frustrated.
Final version of tutorials. If you can't follow that, there's no helping you!!!
We learned valuable lessons as we went through three iterations of tutorials:
- People assume they already know how to play your game. I can't for the life of me figure out why they come with this belief, but they do. Work with it, not against it.
- People don't want to learn (because of the above), they want to play. So teach them one basic thing and set them off playing
- When teaching, we observed the average player's patience is two: Two screens of slides, two steps of interaction, two dialog boxes, then they don't care anymore and want to skip forward
- After playing, people want to learn. There is a correlation between how long they play and how interested they are in learning to play. In the first 2 minutes, they have 0% interest, at 5 minutes they have 10% interest, at 10 minutes they have 50% interest. You have to space your lessons appropriately
- Remove all flowery "in character" text from the tutorial, they want to learn as quickly and efficiently as possible and could care less if a character starts each sentence with "
- They don't want to read, they want to do. Make the tutorial visual and interactive
- Pre-plan your tutorials into the main story/progression of the game. Don't do what we did and try to retrofit it in, it was a lot of work after the fact
Testing with Testflight
As previously mentioned, iOS locks down the devices to which you can install a binary. This requires the unique UDID of each device, registering it through the apple portal, including the provisioning profile at compile time into the binary, and then copying a provisioning profile to the device through iTunes (which provides zero feedback if it was done successfully) and finally copying the binary to the device through iTunes. I can't think of a process more antithetical to the apple "it just works" mantra. So, you can do all that OR you can use testflightapp.com.
Testflight for build distribution during testing; epic win!
With testflight, you send testers (family, friends, enemies) an email link and they go to it on their device. Testflight takes care of finding the device UDID, os version, make/model of device, and sending it to you the developer, installing the provisioning certificate. As a developer, all you need to do is register the device id to your binary. Now you can upload a build (with release notes) to testflight, and everyone you autohrize is sent an email a link to download it. It bypasses all the silly itunes file copying. Testflight's reporting allows you to see who has what installed. Valuable when they start reporting problems as you can definitively say "Oh, that's because you're on a build that was so yesterday! That build was terrible! Install the NEW build, it's wonderful."
Testing with Non-Gamers
Would you rather know about an issue during development, or once it's released? Of course during development!
Testing with people who play similar games as the one you are making is very helpful. You can be sure you are meeting your demographics' demands and they can often make suggestions or give articulate feedback on issues as they have a frame of reference.
However, we found great value in testing with people who have never ever played an electronic game in their life. You know who I'm talking about: your mother-in-law whose only game experience is yahtzee on the dining room table; the friend who didn't know brick breaker was pre-installed on his blackberry.
The official term is blackbox testing. These people can confirm if your tutorials work, but more importantly they do things you never in a million years would do. But make sure you watch them closely, they won't be able to tell you what they did or did not do. Here is an example:
We finished our final bug free never crashes build Dec 15. Over the Christmas holidays I showed a non-gamer friend the finished game. Within 2 minutes of playing he crashed it.
How? He never once lifted his finger from the screen. If you recall from part 3 about our gesture system, it tracks the current finger position every 50ms. Well if the player never ever lifts their finger from the screen, it becomes one giant gesture of 2,400 points after 2 minutes (affecting performance). Even worse, the initial target object of the gesture may be destroyed while waiting for the gesture to finish, resulting in a NULL reference and therefore a crash.
It was relatively easy to replicate and fix once I saw what he did, but I have to admit I never imagined someone not lifting their finger!
How Do You Know When You are Done Testing?
This may seem like a silly question, especially coming from a business software background. In business software you would already have all your test cases written before hand (Right?) and execute them. When they all pass, you know you are good to release. Typically we would then get the customer to test the application in a pilot project, and that is the "real world" test. If it's good, we release.
Well in games, it's different. There is no "customer" to sign off and take responsibility that the application is good, you just have to decide at some point: it's done.
Release too early, and the game will crash or misbehave for customers. Release too late, and you've squandered valuable effort that could have gone into another title.
I was listening to the MIT Open Courseware on Game Design and they asked this very question: how do you know testing is done?
Their answer: When you are out of time.
When you can't take another step forward, you might be done testing.
That seemed like a cheeky answer, but having now lived it, I agree. Now, of course, we made certain all features worked, it was as fun as we could make it, and it didn't crash. (Of course now as I write this we've heard reports of a bug in one of our tutorials, oops!). The game will never be perfect, there is always more to add, more to test. There comes a point where you have to draw a line in the sand and ship it. For perfectionists, this is a very difficult thing to do. I am fortunate that I'm not working solo on this project, both the artist and I together were able to agree the game was ready to go out. That gave me confidence I wasn't deluding myself or just fed up. J For those soloing it, I recommend you ask a friend to be your "quality control" and help give you the thumbs up for releasing.
Judging the Difficulty
I read in the book Level Up: Guide to Great Game Design that the game makers are the worst people for judging the difficulty. So, I knew this going in, but it is still difficult in practice. Obviously the first people that need to be happy are the ones making the game, that is your first quality control gate.
We also tested on young children (3, 8, and 9). Why? Because they'll test for free all day long and they have nothing better to do. (And they are the artist's children). We found that young kids love to play Catch the Monkey, but the mid game was too hard for them. So thinking this was a secondary market, we created an Easy mode that gives the player more energy and reduces the maximum number of monkeys in the field at a single time. The kids loved the easy mode and were able to finish the game, so we were happy.
Later, when doing focus group testing, two of the teenage girls couldn't get past a certain level and were getting frustrated. We recommended they restart the game in easy mode. As soon as they started playing in easy mode they said "Oh wow, this is much more fun."
At this point we had a dilemma: do we make easy mode normal mode, and normal mode hard mode? We did and made all the code changes to reflect this.
Then, a few days later, we fixed a bug in the Level GM AI and found it was working much better than previously. So we flipped it all back to Easy and Normal rather than Normal and Hard.
Now that we have released and friends/family have been buying it, the most common complaint we hear from casual gamers is that it is too hard. When we tell them through facebook to try it on easy mode, they always come back with "Oh wow, this is much more fun." Even post release we're still learning things the hard way!
Here are the key lessons we've learned:
- Make the game too easy, rather than too hard. Too easy can still be enjoyable, too hard never is.
- Casual gamers are not looking for a challenge, they are looking to pass the time. Easy fits within this expectation.
- Don't make the game hard with the ability for the player to opt into easy. Make it easy with the ability to opt into hard.
Releasing to the App Store
After making it through the arduous testing phase, we were ready to release this sucker. This has several steps:
[indent=1]1) Making a build signed for App Store, as opposed to ad-hoc provisioning build
- They don't want to read, they want to do. Make the tutorial visual and interactive
[indent=1]2) Uploading the binary to Apple