Introduction

This article is aimed to be read by people (designers, programmers, managers or artists) who know already what playtests are, what it can bring to game development and how it basically works. If you're not, I can only advise you to take a closer look at this site.

Just one thing before going ahead: playtests ARE NOT functional testing; we are not trying to hunt bugs here. Instead it may be very similar to usability testing as applied to games.

I'll present in this article a few tips for playtest managers out there who are just starting out. I'd like to say that playtests are a matter of people and psychology. Every manager has their own way to drive playtest sessions and there is not a good or a bad way to drive sessions taking into consideration a few rules (such as "don't influence playtesters!").

I've been introduced to playtest management by Pascal Luban for David Douillet Judo, a game which has been developed at 10Tacle Studios Belgium. My task was to learn from his playtest manager experience in order to hold a small playtest in-house structure. For smaller studios, part of the job of a game designer can be to handle such structure, as was my case.

Pascal Luban has done a great job when he was playtest manager on Splinter Cell Chaos Theory Multiplayer for Ubisoft. He wrote several articles on Gamasutra talking about playtesting and more specifically playtests for FPS games.

Splinter cell chaos theory mixes
both FPS and Infiltration gameplay in multiplayer

David Douillet Judo was a mid-size combat sports game for PC, PS2 and GameCube on which development lasted one year. Playtests started 6 months before release and sessions planned for this game gathered about 200 testers for approximately 40 sessions. We are far from Microsoft's playtests with thousand of testers but we are nonetheless using the same tools.

Despite its mid budget, David Douillet Judo
has a very complex animation system and statecharts

You'll find in this article 10 tips to (I hope) achieve better results with playtests and how to organize them in a better way. The tips are based on my empirical and own experience and may diverge from usual playtesting methods but I assume you can figure out what is relevant for you from what's not.

Let's start!

#1 - KNOW THE GAME

The first tip seems obvious, let's take a closer look at it. On David Douillet Judo (DDJ), I was game designer as well as playtest manager and for me the goals in terms of feelings, game experience and target were very clear. But let's just imagine that I wasn't designer on the project, should my approach on the project be different? Of course not!

What I want to highlight in this tip is the importance to understand all the aspects of the game you're playtesting (target audience, feelings, experience that the designers want to provide) in order to provide good and useful feedback to the development team. You cannot be too vague or too specific. As playtest coordinator, you'll get plenty of recommendations from the players. To be able to sort them out efficiently you have to understand the needs of the game.

Playtest managers are a bit of mind readers

To do so you'll have to communicate with the team, designers, players, management and, of course, play the game by yourself and ask yourself: "what are the needs for this game to be a success?"

#2 - INVITE TEAM MEMBERS

Participation of team members inside playtest sessions is an important factor for a successful playtest program.

Team members spot problems more accurately than you do. Nothing personal, it is normal. They've been creating the game and its mechanics for several months. Showing them a new approach of the game when testers are playing may sometimes reveal things that they haven't spotted (such as a missed item in a level, a camera angle that is not very helpful in a specific situation, etc). In order to take advantage of such experience, invited team members should be as open as possible to changes and be prepared for abrupt criticisms from players, so don't forget to brief them before.

Team members will very accurately see and spot if the intended behavior of a mechanism is working as it was designed and will take notes of solutions which can resolve the problem.

After the session, organize a debriefing with the team members to expose and sort out the biggest problems and try to find solutions for each of them. Most of the time team members will come up with clever and efficient solutions. As an "out of the project" person, playtests managers can also give a fresh and new approach for most complex problems.

At the end of the debriefing every important problem should have a solution. A report has then to be validated by the management for solutions to be implemented.

Last but not least, inviting team members strengthens the trust they will have in your work and results. In the end, everybody will be more open to the changes that the playtest department is suggesting if they can trust the people behind it.

Of course you can plan to start playtesting in-house with the team itself, invite team members individually and let them do a session. This way, you'll be able to gather valuable information in a more objective way.

#3 - TRUE, FOCUSED

During the development of DDJ I noticed how it was easy to hide problems behind nice sentences or the development progress of the game. Sentences such as "Oh don't worry this camera will change in the end" or "We will add a sound to cover this action in few weeks".

Don't be blind to problems, instead expose them. If the player feels the game lacks some sounds or music, then say it in your report. Of course if the focus of the session is the camera try to sort problems concerning the session first. Gather other recommendations (such as sounds) in a database and regroup recommendations in families (see tip 6).

If you spot one particular problem and if a player spots the same problem then trust your instinct, take notes and investigate with the team after the session to find a solution.

If you don't, the unsolved issues will haunt the project and impact reviews, trust me on this.

Being a playtest manager is not an easy job; when you spot a problem (design, interface, gameplay, feedback) it also means that you point out a defect in someone's work. If you have a direct contact with the team, tell them how to improve, find solutions together and offer new ways to think instead of simply criticizing. This of course has to be balanced between costs and time.

In the end, your job is not to sell the game to the players; your job is to see where the game can be improved no matter whom has done what or why they did it that way (if it doesn't work as it is supposed to). Major issues will be fixed but don't be too demanding, even with tons of sessions not all problems will be solved.

#4 - RECORD, RECORD, RECORD

Record as much as you can!

If you have money to buy expensive cameras do so because it will save you time when presenting problems to the team. One thing to remember: video is worth many statistics and explanations.

In my case I hadn't much money. I had to find cheaper ways to record things.

I setup a few RJ45 webcams to record the players' movements. I setup one to record the hands and the pad and placed another one in front of the player to see facial expressions. Webcams have a higher resolution than a few years ago and are cheaper than cameras. It has become a good alternative to high end cameras.

Webcams or cameras are very interesting to see how players behave with the pad and how they feel when they play the game. For example you can spot and analyze frustration when a player cannot trigger a combo or see if he is frightened in a particular scene where he should be.

Most of today's webcams have a web server embedded and you can monitor everything on a single screen. That way you don't have to be close to the players (which may influence them). You can even invite team members to watch the session on their computer using a simple internet browser.

I also setup another device, the most expensive in my case, which was an output video stream converter to record what's being rendered on the graphics card. Again, the stream can be watched from any browser using a special ActiveX.

Playtesters' interviews were recorded on an MP3 recorder. It is not an expensive device and it can save you time when you're writing the sessions reports. It may also serve later or after the release of the game as a base to launch a focus group for example.

Don't forget the way you'll store such files. Files space occupation represents quite an amount of gigabytes at the end of all sessions because you have to record each session. AVI and MP3 files can be stored on a local computer and afterwards saved on a DVD. If you're lucky enough and have IT administrators, let them handle this for you.

Even if it's important to record everything, try to find a solution that suits your needs. A simple rule is: you shouldn't spend more time organizing files than doing sessions.

#5 - BUILD YOUR FRAMEWORK

It is important to target the needs of the project and yours too. For example if you plan to make a few sessions (lets say less than 10) maybe it is not worth it to invest into a program to handle testers, as a simple Excel list will do the trick. If you plan to do more sessions, maybe it is worthwhile to ask a PHP specialist in the team to develop a quick web interface so can you can manage a playtesting database.

I started with an Access database, which was the way Pascal Luban worked on his previous projects. I quickly moved to an upgraded version and programmed a MySQL database which was accessed in PHP. MySQL simply offers more flexibility. Web programs such as managers, mailers and schedulers can be easily developed and accessed anywhere using PHP and a remote web access.

For example, you can setup a small website with a web page containing a simple form for new testers who want to join in. You setup an administration section protected with a password to accept or deny new testers. Accepted testers are automatically put in the database and called for next sessions (using a mailer daemon).

Emails also took up a lot of my time. It takes just 2 minutes to write a short email to one person but it takes sometimes a few hours to send invitations to something like 200 testers. You can nearly automate this task by preparing text templates that you will reuse for each kind of email you have to send (subscription, validation, invitations, etc...). A good combination to this system is to create a mailing list or to develop a small mass email program using PHP (you can also use a bulletin board such as PHPBB to deploy your playtest platform. It natively embeds such features).

PHPBB is a tool that you can use to setup your database

The idea is again to avoid losing time with tidy tasks. If you're alone, all those small tasks can take a large amount of your time. As you've already figured out, this is can be critical.

I'm actually developing a small web program to help playtest managers to keep track of sessions, so stay tuned if you're interested...

Playtest Engine is a full management
environment for playtest managers

#6 - KEEPING TRACK OF RECOMMENDATIONS

During sessions you'll receive tons of suggestions and recommendations from the players. It is your job to evaluate if suggestions suit the project or not. Don't reject every weird idea, sometimes some are good and will give the team a fresh look on a specific problem.

You cannot prevent players from making recommendations or suggestions, because they played this game and it was "so f****** great" they will try to push ideas they liked into your game. To be able to discriminate the good from the bad don't hesitate to ask before the session: What are the last games the playtester played? What are the best and worst games they played? And take this into consideration.

Babes are not a (real) valid recommendation...

Once you've selected the ideas which seem interesting, try to organize them by groups and families (sound, art, animation, gameplay, camera, FX, plot, etc...). Organize them by priorities and discuss them with the team and the management to see what is feasible or planned. There is a simple rule to setup priorities: the more a suggestion comes back from the mouth of playtesters, the higher its priority.

Then keep track of the acknowledged suggestions and recommendations and continue to prioritize suggestions after each session. If a suggestion has been implemented in the game, try to evaluate its impact on the playtesters.

There are links between recommendations and it is not unusual to see that if a recommendation is pushed into the game, another one may become a priority or just fade away. You'll have to constantly update your database of recommendations in order to have the latest version to present to the team and management.

#7 - CREATE A "HOME" FEELING

It is really the most difficult goal to achieve. How can somebody feel like being at "home" when every movement, mistakes and achievements that they make are recorded and analyzed?

Home sweet home

The first point to do is to be pretty much clear about what the playtesters are going to do, what kind of information you'll gather and how you'll do it. Secondly, explicitly say that mistakes are not something bad, that people are here to find ways to improve the game and that you know that the game is not perfect. Don't forget to mention to the players that saying "everything is great" doesn't really help the team behind the game.

I often say to playtesters "imagine that you're with friends and you try to review what's best and what's worst in the game you're currently playing". It makes them more comfortable with criticizing the game and its content.

Once this first step is done, it will be easier for the players to concentrate if you're not standing too close to them. Try to keep distance and if possible stay out of the room (see point 4).

If you have more than one playtester at a time, don't hesitate to put them close to each other. It is a privileged moment for you to see if the playtesters communicate about the game during the session (strategies, controls, behaviors, etc...).

If you plan to give a gift at the end of the session, don't mention it. If you do, playtesters may think that they will have to give good evaluation to the game to get the gift. Also don't offer gifts which are more valuable than a few goodies or you'll get tons of subscriptions by people only interested in gifts.

The last point is the most difficult to achieve because it also requires money. To make playtesters even more "at home" and comfortable first look at offering free drinks before and during the session. A small fridge in the playtest room is even better. They will be able to take a drink whenever they want (and not have to ask you for it). If the game runs on a console, you can also trade the good old desk and chair for a couch. Make it clear with the testers that they can take a break whenever they want. It is very important for concentration and a good evaluation of the game.

Finally don't forget to decorate the room with posters, goodies, and artworks to avoid the "hospital room" syndrome.

#8 - THE EARLIER, THE BETTER

This is a very simple tip here, start as soon as possible.

The team will try to delay the playtest process as much as possible for many reasons: stability, bugs, gameplay or polishing. Try to explain why playtests can be a good thing for production, why they are part of the production and why we need to start ASAP. Remember, in the end the team is making a game that needs to be a playable and fun experience. Playtesting is one (amongst others) accurate tool to see how playable and fun a game is.

If the management or the team say that the game is too early in development for playtests ask them if it is easier to change/remove/add a prototyped system made in two weeks or a three months long feature?

Time travel is not yet possible...

To make management and team more comfortable with starting early, you need to brief testers about what they're going to test. If correctly briefed, playtesters will overlook the graphics and evaluate the gameplay and fun potential pointing at eventual problems. In this particular case prefer casual gamers or pro-gamers, they don't look too much at graphics quality.

From my experience, agile methodologies (such as Scrum) are better suited to support playtests while in production. Agile methodologies are more flexible and production can be more easily adapted to results from playtests. This can be a little bit more difficult (but not impossible) in a waterfall system.

If you start early in the development be sure to set clear goals for sessions (see point 3).

#9 - GAIN THE LOYALTY OF PLAYTESTERS

When playtesters are coming to the studio some of them have stars in their eyes. It is especially true if the franchise is well known (and expected by the community) or if the studio has already released a few good games. I'm pretty sure that Halo 3 got tons of playtest subscriptions.

But then how to convince someone to come when nobody knows your studio or if the management told you not to announce the game? You'll have to win playtesters' loyalty.

First of all you may have to go to each game store in your city and talk a bit about the studio and the game (without revealing the name, but you can tease people explaining the genre and what people will test and how they can impact the development of a game).

If you're doing in-house playtesting, people who live close to the studio are the best choice. They don't have to travel a lot to test the game and can come back if needed. They will also talk with their friends about what they did in the studio, if the experience was good, be sure that friends will subscribe and come too.

Try to select testers which have an interest in the genre. If the game is still in an early version explain exactly what that means (most of gamers don't have a clue as to how games are created).

If a tester doesn't like the game, don't blame them or neglect them, you have to be prepared to handle this kind of situation. Instead ask him why he doesn't like the game and note his comments. Even if his feedback was negative (and if you planned to), reward him.

The idea behind all this is to make the playtest session a good experience (even if playing the game was not a good experience). If possible try to organize a quick studio tour and present people behind the game. Again, if possible, put playtesters who came more than twice in the credits menu of the game.

This way people will remember a pleasant experience and will share this with their friends.

Having such an attitude will make your playtesters database grow without any new advertisements, which is time consuming. It is saved time that you don't waste searching for new testers.

#10 - TARGET LARGE REFINE LATER

When you're searching for playtesters ask everywhere: local store, schools, universities and websites (even non game related). Gather as many subscriptions as possible.

The second point is to make sure that people are interested in the playtests. Send them an email and explain exactly what playtests are, that they are going to be invited, what they are going to do etc.... Don't forget to mention that even if the person is in the database it doesn't mean that they will be selected. Selection depends on the criteria of the session. But it is important to say that being in the database also offers the right to come for another session and even for another game.

Shotgun is better suited than the railgun

It is important to target large because you don't exactly know peoples habits. Maybe a father will see an ad in the newspaper and will say to his son: "Hey did you see that this game studio seeks testers, you're always playing with your PSP! Why don't you go?".

Targeting large is also a good tool to give you a large user database. You don't know what the future is to be made of. The previous game of your company was marketed for casual gamers but the studio decided to launch a new game targeted towards hardcore-shoot'em-up-strategy gamers. If you have a large user base, you'll be able to start playtests in a week.

Bruno Urbain is a game designer at 10Tacle Studios Belgium. Bruno has been in the industry since 1994; worked 10 years in mod development and 3 years as a professional game designer. He's worked on titles such as Medieval Lords, David Douillet Judo and is currently working on the recently announced Totems. You can find more about his work on his personal website http://www.bruno-urbain.com