• entries
    206
  • comments
    217
  • views
    224509

Links: Beta Testing

Sign in to follow this  

383 views

Something that has been weighing on my mind for awhile is the question of how to run a decent Beta test (or any sort of test) for an independent, single-player RPG. Specifically, my RPG, still titled "Untitled SENG Game". I'm approaching the point where I'll have some gameplay prototypes ready, and I'd like to have some unbiased people take a look at them. As it turns out, I've seen a couple of decent posts on beta testing, so I thought I'd share.



First, Amanda Fitch of Amaranth Games (http://www.amaranthia.com) posted her guide to selling indie games:

http://www.amaranthia.com/downloads/How-to-Sell-your-Game.pdf

This is a nice, short, nuts and bolts guide to the details of actually selling your game once you've made it, with a short section on Beta testing.



Next, Jeff Vogel (Spiderweb Software - http://www.spiderwebsoftware.com/) has a really nice, detailed blog post (on the Bottom Feeder - http://jeff-vogel.blogspot.com/)on running a Beta test:

http://jeff-vogel.blogspot.com/2009/08/beta-testers-getting-them-keeping-them.html

This is really helpful and worthwhile; the man has done this sort of thing before!



I have two experiences with running a Beta test for an indie game. First, I ran a long and extensive test for my play-by-email game Atlantis (http://www.prankster.com/project/index.htm). I had several hundred people playing, which made for an excellent testing atmosphere, plus two other things really helped. First, as a server-based game, I had complete access to the game as it was going on; I could see what orders the players were issuing, complete reports on their positions, etc. Basically I had all the data (and more) that I could want for doing testing. Second, as a multi-player online game, the players were already used to lots of discussion, with mailing lists, a weekly newsletter, etc. Getting feedback or bug reports wasn't a problem at all; the problem was distilling all the information into something helpful!



My other experience was testing my free SENG demo "To the World Tree" (http://www.prankster.com/ttwt/index.htm). My experience here couldn't have been more different from Atlantis. Since it was just a demo, I did not set up a formal testing program; instead I just released it and asked people to send feedback. This was a mistake; I don't really know how many people played, though it was downloaded thousands of times, who knows who actually played the game? I got a few really detailed feedback forms with lots of helpful information. I got more (but still not a ton) of limited feedback; this feedback could generally be distilled into either "This sucks!" or "Attaboy!". But mostly I just got silence.

So, I have two thoughts going forward as far as testing. The first is definitely to create a small but active community for testing. Only allow the testers access to the game bits (don't just release a demo into the wild), and really strongly try to interact with them (and even get them to interact with each other). I suspect some combination of an announcement mailing list and a forum is the best way to go.

Next, some sort of real, hard data from the game would be nice, too. I don't want to try to get super-high-tech (because that would take a lot of effort), but perhaps the game could periodically offer the player a log file that he could email to me or something. Maybe a web-form would be even easier. This log file wouldn't help much with bug-fixing, but if I logged the right things, I could get a good idea of how the player is approaching the game. Which class he is playing, which quests he is doing at what levels, what monsters are particularly easy or difficult, etc. That kind of detail would be really nice to have, and it is pretty hard to extract from the player directly, without actually watching him play.

Those are my thoughts; if nothing else I really recommend you read Vogel's piece for good solid advice. Depending on my work plans, I may be getting into some sort of testing program this fall.
Sign in to follow this  


3 Comments


Recommended Comments

Wow, didn't know you were behind "To the World Tree" -- unfortunately I was a contributor to the silence you received. I thought it was a great demo but didn't have much feedback at the time. Sorry for that in hindsight :(

Before I offer my advice, two concessions:
(1) I've never released an indie RPG -- but I have released indie games and a mainstream RPG :)
(2) I've not made near the bucks that Amanda or Jeff have made, and thus don't have as much data to support my claims :)

With that out of the way... Do it in two phases. Keep the first phase tight, very tight. Around 10 players max. You'll need quality testers that will finish the whole game; hopefully multiple times. On this scale, the data you will get won't be overwhelming and will help find scripting errors, major gameplay bugs, and other application crippling issues. It is hard to find 10 really good testers like this at random; I would suggest paying a small QA company or individuals for this service -- it may save you much more in the long run.

The second phase is the shotgun period. This is where you take on a much larger pool of testers (where it will be difficult to manage all of the incoming data as a single person) and you will find a lot of the outlier bugs: resolution incompatibilities, hardware issues on ancient or esoteric or even fairly common machines, and you will also get flooded with more minor bugs -- which you can elect to fix or not fix based on the time you have.

For a large scale game like an RPG that is more or less very much open ended in how you choose to play the game and progress, I have found this approach lends the best results.

Whatever approach you take, I hope it works out for you! There's nothing better than a good indie RPG and that market has expanded immensely in the last few years.

I'll also throw in my hat as a beta tester if you need one :)

G'luck!!

Share this comment


Link to comment
Quote:
Original post by shadowcomplex
Wow, didn't know you were behind "To the World Tree" -- unfortunately I was a contributor to the silence you received. I thought it was a great demo but didn't have much feedback at the time. Sorry for that in hindsight :(


No big deal on the feedback; I blame myself for the way I ran the test. Glad you liked it, in any case! The engine is much better now than it was; better UI, more character options. There's even less of a game associated with it right now, though.

Quote:
With that out of the way... Do it in two phases. Keep the first phase tight, very tight. Around 10 players max. You'll need quality testers that will finish the whole game; hopefully multiple times. On this scale, the data you will get won't be overwhelming and will help find scripting errors, major gameplay bugs, and other application crippling issues. It is hard to find 10 really good testers like this at random; I would suggest paying a small QA company or individuals for this service -- it may save you much more in the long run.

The second phase is the shotgun period. This is where you take on a much larger pool of testers (where it will be difficult to manage all of the incoming data as a single person) and you will find a lot of the outlier bugs: resolution incompatibilities, hardware issues on ancient or esoteric or even fairly common machines, and you will also get flooded with more minor bugs -- which you can elect to fix or not fix based on the time you have.


I've got a fair amount of experience working in a "real" software company, and what you're suggesting is much more inline with what we'd do there. The first step would be in-house testers; employees or contractors tasked with finding bugs and design flaws from the early stages of development. The second step is called a beta-test, designed to find bugs across a wide range of hardware and usage scenarios. By the time you do a beta-test, you typically don't have time in your product cycle to address major design flaws or oversights, so you better not have any.

There's even a third step you don't mention; something like "usability testing". Here, early in the development process, you get a set of knowledgable but un-involved users to try out your product, and you watch them do it. This is great at turning up design flaws and so forth that your core team has become blinded to from constant exposure to the product.

As an indie, my thought has been to do in-house testing myself, and hopefully the scope of bugs I miss is limited through good development practices and design. I can do some basic usability testing with friends or family. And the beta test is as above, though it may turn up some larger design flaws (due to the lack of in-house testing) which might need to be addressed, possibly leading to a second beta.

Quote:
For a large scale game like an RPG that is more or less very much open ended in how you choose to play the game and progress, I have found this approach lends the best results.


Agreed, with the caveat that an indie has to be really clever to deal with his lack of resources.

Quote:
Whatever approach you take, I hope it works out for you! There's nothing better than a good indie RPG and that market has expanded immensely in the last few years.

I'll also throw in my hat as a beta tester if you need one :)

G'luck!!


Thanks! I will bug you when the time comes.

Geoff

Share this comment


Link to comment
This was a very informative journal entry for me. I've always found it difficult to find quality testers (people that have played the genre before, that have an interest in the game, and that are willing to give me constructive feedback). With iPhone development the beta testing has become almost completely impossible due to the deployment restrictions. I usually end up putting it on a friend's ipod and trying to watch them while they play. But the hardest part has definitely been gauging the difficulty of opponents, bosses, etc.

RPGs are one of my favorite genres so I follow your posts quite often. Everything is looking good so far. Attaboy!

That said, count me in for beta testing and I'll dig out my PC just for you.

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now