Game QA & Testing Chapter 3
The Many Faces of Testing
by Luis Levy and Jeannie Novak

Testing Disciplines

As you learned in Chapter 1, testing started as a couple of people making sure the game worked. From there, it evolved into an essential part of a game’s lifecycle—the “be all, end all” of a solid title. If your game had shoddy QA, sales would soon nosedive—and retailers wouldn’t be happy. The testing disciplines listed in this section are areas of knowledge within testing—in contrast to testing techniques (which will be discussed in Chapter 6).

Balance Testing

Balance testing ensures that gameplay is fair to both the human and AI player alike. If one is too powerful, there’s something wrong. In single-player games, balancing involves making sure that the “easy” level is not too easy, “hard” is not too hard—and “medium” offers a nice, gradual rise in difficulty. Multiplayer games need to be balanced as well. Maps need to be neutral in their design, weapons must be equal in power (if not, the characters wielding them should be), and spawn points need to be placed fairly. The truth is that without balance, all matches are skewed and no victory is fair—or fun. This is so important that developers sometimes spend months balancing the gameplay!

Balance testing demands acute attention to detail. At the end of the day, it’s the feel of the game (and your notes) that will help the developer fine-tune the gameplay. It’s a fun job—exhilarating sometimes, but a very serious task nonetheless.

How to Balance a Game

Here are some guidelines for balancing a game during the testing process:

Make sure the code is stable. Crashes and freezes will make balancing impossible.
Enlist an equal number of testers for each side. Make sure their skill levels are matched by members of the opposite team.
To balance weapons, load a neutral map. To balance a map, have everyone equip weapons of the same class.
Play for at least one hour, making notes on the strengths and weaknesses of each weapon class. Write down the score as well, if a scoring system has been implemented.
When the hour is up, have everyone switch teams. Do the same with the new set of weapons.
Start an impromptu discussion. Write down opinions, observations, and final scores.
You should have a hypothesis now. Check with the lead to see if you have enough time to investi-gate your hypothesis. If the lead suggests it, arrange everyone into different task forces and test it. (Task forces will be discussed in more detail in Chapter 6.)
Chances are, you now have a good idea of how each weapon feels. Communicate your findings to the lead in written form.

Once developers come up with a new build, you’ll have to do this all over again. In Call of Duty 3, the producers tasked the team with playing a lot of Halo 2 at home for balancing. After this was done, the devs changed all weapon values in the game to Halo 2 levels. Gameplay became arcade-ish, with several hits before someone died. Once the entire team despised the new values, the devs interceded again and brought them closer to CoD levels—not quite like CoD2 but closer than Halo! The result was a game easier to play than CoD2, but not as forgiving as Halo—which made it fun. Ultimately, this is the purpose of all balance testing. If you see players describing your game as “fun” on the Internet, it’s a job well done!

Compatibility Testing

Exclusive to PC, compatibility testing is the process by which the game is tested with multiple hardware configurations. The objective is to make sure the game is fully compatible with parts and peripherals found on the market. Testers need to be proficient in PC assembly and troubleshooting, since the job consists of installing multiple pieces of hardware and seeing how the game runs with each one of them. Compatibility testing is much more technically demanding than balance testing. Installing GPUs (graphics processing units, the high-performance video cards used for 3D games) and RAM modules is not for the faint of heart. Even background applications can have an impact on how a game runs. Knowing how to perform compatibility testing is a somewhat rare skill, and most testers will never need to learn it. However, in order to advance your career, you might have to tackle compatibility testing before moving on to other things.

Most games made by Blizzard (e.g., Diablo, Diablo II, World of Warcraft, StarCraft) are virtually bug-free because Blizzard does not mind pushing a launch back six months in order to present a polished product.
Floyd Billings, (Assistant QA Lead, Sony Online Entertainment)
Audio Compatibility Testing for Mobile Games
In particular, music and sound for cell phone games present a challenge because every phone has a different instrument set. What sounds good on one phone can sound horrible on another. Our phone game projects require testing on a lot of devices to ensure the greatest level of accuracy.
Eric Doggett (Owner, Moondog Media)
I am currently working on a 14-game project for one company. The games will be ported to over 15 different mobile phones, each with their own audio file types, size limits, and speakers. I had to audition each phone and build my own playback system comprising two ½-inch speakers configured to play back directly from my workstation. When speakers are this small, much of the bass and treble are stripped away and the sound is extremely distorted. There is a lot of work involved in testing these devices and making them sound good!
Ben Long (Composer & Sound Designer, Noise Buffet)

Soaking

“Soaking” sounds like cooking—and in a way, it is. In game development, soaking is a sub-discipline of testing—one where the tester doesn’t really do anything but leaves the console or computer running for extended lengths of time. For example, a tester might leave the game on the title screen for three days and then verify whether it crashed or not. Soaking is necessary because sometimes memory leaks or rounding errors might hurt the game’s stability in the long run—which would be undetectable during normal testing. Soaking testing is usually done by a lead tester.

Compliance Testing

Compliance testing is the one testing variety everybody seems to hate. But what gives compliance testing such a bad rap? Before a game is sold in the market, it first needs to go though certification; this process is conducted by the hardware developers themselves (such as Sony, Microsoft or Nintendo), who need to be absolutely sure that a game follows established guidelines before it is allowed to be sold on store shelves:

Sony has what is called TRC (Technical Requirements Checklist).
Microsoft’s is similar, but not the same: TCR (Technical Certification Requirements).
Nintendo uses the same term it did for Super Mario Bros. in the 1980s: Lot Check. (See Chapter 1 for more details.)

Each compliance checklist is composed of several different basic categories, such as the warning that flashes on the screen if you don’t have enough storage space. This warning must follow certain guidelines or the game will fail once it reaches the console manufacturer (which we usually refer to as “first party”).

Nintendo has a really good reputation for having games that appear to be bug-free. Super Mario Galaxy on the Wii was not only a great game, but it also showed tremendous polish and a dedication to making sure a weird bug never took you out of the experience.
Josh Bear (Chief Creative Officer, Twisted Pixel Games)
RocketBowl Microsoft Certification
We developed RocketBowl for Xbox Live Arcade last year, and the testing phase was a grueling 10 weeks. Microsoft certification requirements are huge (250 pages of them), especially for multiplayer games. Even with all of the testing, we failed certification the first time (mostly due to translation issues) and had to resubmit. Overall, we learned a lot about how to make a solid product for XBLA (Xbox Live Arcade).
Justin Mette (President, 21-6 Productions)
Grand Theft Auto IV: The Rockstar of Compliance Testing
Rockstar’s Grand Theft Auto IV was a very clean Format QA [another term for compliance testing] submission from the beginning. This high-profile title had its release date slip—which most likely contributed to the devel-oper’s ability to ensure that bugs were minimized and that features were thoroughly tested in-house. Additionally, it should not be overlooked that the large budget most likely played a key role in ensuring that project management could devote necessary resources to do the work—and do it right. Finally, with a simultaneous multi-platform release date, this increased total headcount—and I’m positive more eyes were working the product as a result of this marketing strategy. In conclusion, it cannot be stressed enough that top notch software engineers and supportive project management undoubtedly played the single most important role in this title’s tremendous success.
Jerome Strach (QA Manager, Sony Computer Entertainment America)

Other common compliance requirements:

The game must not display any religious images
The game should pause once the controllers are not attached
Before formatting a memory card or a hard drive, a message warns the user that all the data will be erased
All copyrighted brands or names are acompanied by their respective copyright or trademark signs
If data gets corrupted, the game warns the user and suggests reformatting the memory card or hard drive

Testing for compliance gets repetitive quite easily. Can you imagine testing a PS2 memory card for two hours, following very specific instructions? There’s simply no playing involved in something like this; it’s hard work—that’s all. Combing the game for these items is very difficult; very few testers are actually able to go though the hardware manufacturer’s requirements and find all the compliance bugs. Compliance testing is a unique skill and one developers and publishers are known to seek out in new hires.

Compliance Testing for Gun
Luis was once selected among a large group of testers to learn compliance testing. This happened in the lull between two projects. Each one of the testers was provided with old builds (preliminary versions) of Gun and we would spend days going though the compliance checklists, trying to get the hang of it. While some of the items were fairly simple (e.g., “the game must go to pause mode once the controller is disconnected”), others could have been classified as “hardcore” (such as the PS2 network adaptor, which required a CD driver)!

Localization Testing

Localization testing is a reality because we live in a global economy. “Localization” involves converting a game from one region to another, and it usually includes translation. A game might be made in China with the help of Irish and Australian developers—or maybe it was made by someone in Los Angeles, but it was exported worldwide. More com-monly, games are made in Japan and Korea and end up on American shores. For example, Konami and Capcom are famous Japanese developers, and Gravity Interactive is a name brand in South Korea. Many of Nintendo’s core games first see life in Japan, and then they are localized for North America. Once these games get exported overseas, they need to be translated. However, these translation jobs are sometimes full of mistakes. This is a bad sign for a foreign game, especially if the game in question is striving for some kind of dramatic impact.

On the PC side, Blizzard rules the roost on polish. Diablo, Warcraft, Starcraft, World of Warcraft: All of those games were polished to an incredible level for the PC.
Starr Long (Executive Producer, The Walt Disney Company)
Localization Testing in Call of Duty 2
Usually, games are tested once they arrive in the US, but Luis had to do localization testing on Call of Duty 2 once it went to China and Japan. His character would have to keep dying, over and over again, so that Luis could compare the “death quotes” on the screen with the correct Chinese or Japanese characters. This was pure torture—and he’s glad he never had to do that again! Still, it’s a job that must be done—and there’s always somebody that enjoys this kind of task. Currently, Korean MMOs are the ones that must go through careful localization—since mainstream Japanese games rarely have this problem anymore.
All Your Base are Belong to Us”: Zero Wing’s Claim to Fame?
The most famous example of bad translation can be found in Zero Wing. “All Your Base Are Belong to Us”—a line from the intro of the Sega Genesis game—inspired a slew of media homages (including techno dance tracks, t-shirts, and April Fool’s jokes). Even YouTube posted the phrase “ALL YOUR VIDEO ARE BELONG TO US” when it was taken down temporarily for maintenance.
As an example, here is the full dialogue seen in the game’s intro: Narrator: In A.D. 2101, war was beginning.
Captain: What happen?
Mechanic: Somebody set up us the bomb.
Operator: We get signal.
Captain: What!
Operator: Main screen turn on.
Captain: It’s you!!
CATS: How are you gentlemen!!
CATS: All your base are belong to us.
CATS: You are on the way to destruction.
Captain: What you say !!
CATS: You have no chance to survive make your time.
CATS: Ha Ha Ha Ha ....
Operator: Captain!!
Captain: Take off every ‘ZIG’ !!
Captain: You know what you doing.
Captain: Move ‘ZIG’.
Captain: For great justice.

Playtesting The “fun testing” seen on television ads is playtesting—first mentioned in Chapter 2. Contrary to productivity software such as Microsoft Office, games need to be fun to play. To simply “work as intended” is not enough; game designers must tap into what is known as “the fun factor.” In a Gamasutra article entitled “Secrets of the Sages: Level Design”, The Levellord from Ritual Entertainment, explains:

“There are no defined rules for fun, and the only way to ensure the Fun Factor is to playtest. The easy part about adding the Fun Factor is that most all of us have the same concept of fun; that is, if you the game developer think it’s fun, then the game audience is likely to think so, too. The Fun Factor is not transient or ephemeral, either. It should survive countless trials and tests and still be entertaining in the end. This is the only way to ensure that a game is fun—to play test it over and over.”

Making a game fun to play takes a lot more than good intentions and magic dust. It takes guts, determination, and exemplary teamwork. Likewise, playtesting is not something you do to pass the time and have a couple of laughs. The best playtesters can divide themselves into two personas: the player, and the professional.

Player: Always ready for the next thrill
Professional: Always watchful of gameplay mechanics (e.g., navigation, aiming, targeting, interaction, behavior, physics, artificial intelligence, goals)

A game can only be great when playtesting is taken very seriously by everyone in the team.

Nintendo’s “Fun Factor”
Lots of developers such as Blizzard and LucasArts demonstrate excellent quality assurance over the years, but Nintendo in particular seems to have taken control of this from the very beginning. Nintendo’s games have always exhibited a fine gameplay balance, a “fun factor,” that often proves elusive in the industry. This emphasis on balancing and polishing ga-meplay, by definition, leads to top-notch quality control. If you’re focusing as hard as possible on making the game fun—and testing and retesting it—you’re bound to uncover and address any significant bugs. Along the way, you’ll gain enough control of the development process so that new bugs aren’t introduced at a point where they may not be found later on.
Jamie Lendino (Composer & Sound Designer, Sound For Games Interactive)
Playtesting in Call of Duty 3: Get that Bike Right!
Playtesting involves going over all possible varieties of play and making sure all of them are fun. This is serious work: All it takes is for one faulty game sub-system to ruin the whole experience. We’ll give an example of Call of Duty 3, a game in which Luis worked as a production tester. Call of Duty 3 changed the CoD series forever with the introduction of vehicles. Until CoD3, all the vehicles in a level were mere decoration—accessories for the level design. With CoD3, jeeps, bikes, and tanks became not only drivable but essential strategy elements.
However, adding vehicles to a first-person shooter (FPS) is more complex than it might seem at first. When you drive a jeep in CoD3, the last thing you want to be is a sitting duck. With a bike, you want to avoid becoming a sluggish moving target—and the tank must be maneuverable enough to cram through tight European villages such as Poissons.
When fine-tuning the bikes, Luis decided he’d do everything he could to make the bikes fun. (Luis is a racing game fanatic, so he figured he was the one for the job.) There were two different bikes for the US and Germany. At first, they were both quite slow—and their handling was sub par. In order for the bikes to be effective strategically, they first needed to be drivable—offering a degree of control to the rider. Bike crashes in real life are deadly, but bike crashes in-game can bring defeat and certain humiliation.
As a production tester, Luis would spend a lot of time driving the bikes fast, jumping, hitting obstacles, and attempting to run his enemies over. When a big problem was spotted, he’d write to the producers in detail—explaining why the current suspension setup was ineffective or “boring.” Soon enough, the programmers made the bikes faster. Later on, they made drifting possible—with a tail-happy rear-end. These two touches of gameplay allowed the bikes to be effective and fun, pushing the fun factor way up.

Usability Testing

Usability testing is a fairly recent advance in software testing—surprisingly so, considering its importance in the player’s satisfaction. Microsoft is a big fan of usability testing—having learned the hard way after users kept complaining about how Windows machines were so much harder to use than a similar Macintosh system.

Usable means “easy to use”—not “easy” as in difficulty level, but intuitive to interact with other characters, use items, find your way, or drive vehicles. If a certain mechanism in the game makes no sense at all (e.g., you can’t aim and move at the same time), this could be written as a usability bug. Perhaps the game was designed that way and nothing is really broken. It doesn’t matter: Gameplay mechanisms need to make sense and be easy to use. You should worry about beating the game, not the control scheme. Energy spent otherwise takes away from the fun you might be having otherwise. Since it is responsible for visual control and feedback schemes, a game’s visual interface is scrutinized during the usability testing process. Although uncommon for QA (which is used to catch more superficial bugs), all testers end up having to deal with usability issues. You may have even had to deal with them yourself—in games you’ve purchased!

We’ve discussed several areas of knowledge within testing that are considered “disciplines”—including balance, com-patibility, compliance, localization, playtesting, and usability testing. Understanding these disciplines will help you become a well-rounded tester. As you become more skilled as a tester, you will be able to apply advanced techniques (discussed in Chapter 6) to each of these disciplines.

As you can see, game testing is many times more complex than the brainless ads on television might suggest. While testing a game is a skill easily learned, understanding the game industry and how it relates to each variety of testing is a much bigger challenge. Where QA testing has a “brute force” aspect to it (after all, you’re banking on a large number of bodies), production testing might be almost Jedi-like—with a couple of ace testers that are the only ones capable of cracking tough networking bugs. In the next chapter, we’ll learn what steps need to be taken before you begin testing.

Chapter Review

Play a game for at least 4 hours—taking notes on the various bugs you find. Categorize each bug with an appropriate severity level (low, medium, high, or critical). What sorts of bugs seem to be higher priority than others? Is there a pattern associated with severity level (e.g., location in the game, discipline, frequency)?
Go to BetaWatcher.com and join an open beta test for an upcoming game. Maintain a journal during your testing experience and report your weekly progress. When the test has completed, discuss what you learned during the process.
What are the testing varieties discussed in this chapter, and how do they differ? Selecting one of these varieties, and then test an existing game based on that variety’s criteria. Provide a list of im-provements that should be made to this game based on the variety you chose.

Contents

	Page 1
	Page 2
	Page 3

	Printable version
	Discuss this article