Sign in to follow this  
titanandrews

MMO load testing

Recommended Posts

Hi, Does anyone know what tools are used in the industry to do load testing for MMO games? I googled "MMO load testing" and it didn't turn up anything useful. I'm sure companies must perform load testing on their servers so they can know how many users they can support. Does anyone have any insight into this side of the industry? many thanks, B

Share this post


Link to post
Share on other sites
Any reasonable measure of load must come from user activity - therefore I expect 99% of the tools for this are created in-house, and based on their own client.

Share this post


Link to post
Share on other sites
You might want to collect samples of user activity, enough to generate reasonably accurate statistics.
For instance, record how many movement packets are sent per second, jumps, actions, and of which size.
Then code a bot that generates traffic based on those statistics and connect several of them to the server.
Depending on the quality of samples collected that should give you an idea of the amount of traffic the server can withstand.

Share this post


Link to post
Share on other sites
Free MMO Beta Stress Test!!!!!

Best way.

Determining required bandwidth is somewhat "trivial". But any kind of stress testing needs to be performed in real world.

Consider this: You determine that your login server can handle 900 requests per minute.

But in real world, your players will be hammering it before it opens. Once it opens, you'll get 1500 requests in a second!

Gameplay testing. You have determined that your zone handles 40 zone transfers per second, and that in zone, it supports 80 players, each seeing other 79.

Great. Good performance. But then, 150 players get into zone, not everyone sees each other. Within next 30 seconds they get together and (150, each sees ~70), and decide to raid a village in next zone. As they move, zone transfer becomes a bottle-neck. Players start lagging, and respond by clicking repeatedly. This increases your input load by 4 times. This causes latency, some players lose connection. Some connect back, require full state synchronization, others are lagging even more, so they keep spamming the server. Next, latency spikes occur, and you need to start performing rollbacks and resyncs. This in turn increases load, and the whole thing spirals into oblivion.

Artificial load determines some values. But ultimately, real player behavior is hard to simulate.

The worst problem when estimating supported loads is the death spiral. This has been studied extensively for HTTP servers. It doesn't matter what server does until it hits 90% load. Then, the choice of algorithm for handling priorities becomes crucial.

One strategy for this is fastest-first. Always serve fastest clients first, and let slower ones wait. It turns out that this is more effective, although intuition would say that fastest clients will generate more traffic, simply because they get more attention.

The final question also becomes: What is "playable". What if your FPS MMO turns into a 1 update per 5 seconds? What if everyone gets a 5 second lag spike every 9 seconds?

How much do you consider tolerable? There's many informal criteria to determine that, but as is the case with bigger engines, playability is determined by actual players. For some games, the audience might tolerate 1 second RTT. In others, it won't accept 150ms RTT. Can this be covered up with interpolation? Or will that lead only to worse experience?


The goal will ultimately not be: "184.7 players" or "over nine-thousand players". Stress test will determine when gameplay experience degrades. Sometimes it might be 55 players, on another occasion 100.

So in the end, server will likely need to limit number of players, either dynamically or through some pre-set criteria, or degrade the fidelity, by reducing player's AOI or reduce number of updates. Or granularity of them.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Free MMO Beta Stress Test!!!!!
Best way.

While beta is a good place to get the final testing done, I can't imagine not doing most of it in the alpha builds.

When I did a quick test of the scalability of my client I posted it in the IRC channel and turned off the 5 client per IP. One of my friends in NL just kept opening them (flash client) and I had the client set up for unrealistic input and the server was set to full state updates to see how stressful things might be if the server goes live after maintenance or something and gets pounded by tons of users. Inconclusive since I already imagine it could handle 500 users. I've never had much luck with users stress testing things.

Since input is normally never a problem. Just get a separate computer and randomize delays in it's packets (throwing random garbage packets too). And have it do things that might happen in the game. Like randomly have the client move to different locations. Then if you need more testing just send it to people to run. Then you get the benefit of intensive activity for less alpha testers.

This might seem very obvious but you should be taking a lot of statistics. I use a C# console app so I just output server info every half a second then clear the screen and do it again. Things like average sent in and out average latency, min and max latency, total bytes in and out on average for clients. And tons of other data (like number of players). Also logging and such to quickly make graphs for later on. (peak hours or zones).

I have been designing my map editor so it's connected into my live server cluster to monitor things. Can be useful in tracking hot spots graphically. Basically so you can easily see, "hey everyone is collecting near that place and that's why zone 4 kept peaking at 8 PM".

Share this post


Link to post
Share on other sites
Thanks for the replies guys!

So I guess what I was thinking was some sort of tool that could "simulate" game play for n number of virtual players. Basically you could describe the protocol to this tool as well as define certain other parameters and the tool could automagically run through all sorts of scenarios. Perhaps there would have to be some sort of AI involved in this. From what you guys are saying, such a tool does not exist and pretty much load testing is done with real users in beta testing scenario. Maybe a tool I am thinking of is just inconceivable?

-B

Share this post


Link to post
Share on other sites
We ripped out the renderer of our client software, and ran 20 "clients" per linux machine on a fairly large cluster (using the same clustering we already were using for servers). We wrote LUA scripts to exercise different actions -- walk here, buy a vehicle, get in, drive over there, get out, ...

Share this post


Link to post
Share on other sites
If you want to get a more accurate, reliable test, your best way to go would be to basically make bots to play your game. You can then log in with the normal client and play around with the bots and see if you can support adding more bots, less bots, what bot actions cause the most server lag, etc. The more life-like the bots, the more realistic the results, though also the more work it'll take of course.

If you don't want to put in that much work you can just use some statistics you get from having people play depending on their location, amount and actions.

Share this post


Link to post
Share on other sites
Quote:
We wrote LUA scripts to exercise different actions


Were these actions that you could repeat over and over again in a loop or did the logic on the server prevent this? I would think to gather statistical data you would need to perform actions several times on each game instance. For actions that you can only do once during the game, I suppose you have to reset the state back to a point where you can repeat the test? How did you handle this?

-B

Share this post


Link to post
Share on other sites
Quote:
Original post by titanandrews
So I guess what I was thinking was some sort of tool that could "simulate" game play for n number of virtual players. Basically you could describe the protocol to this tool as well as define certain other parameters and the tool could automagically run through all sorts of scenarios. Perhaps there would have to be some sort of AI involved in this. From what you guys are saying, such a tool does not exist and pretty much load testing is done with real users in beta testing scenario.


I think what was actually said is that it's application specific. To get real figures, you need real players. To get passable figures, you'll still need to use your own client or something based on it. Merely knowing the protocol isn't enough. Imagine we're talking about chess. You can easily come up with a protocol to send a move and receive a move. But the subsequent moves you make depend on the state of the chessboard, so you need a local representation of the game, as well. And that can't trivially form part of any standard tool.

Quote:
Maybe a tool I am thinking of is just inconceivable?


In theory, you could build up something that would allow you to plug in a network protocol and to have some sort of local game representation. But once the user is forced to provide all this, the tool barely gives any additional benefit on top. Why not just build the stress-testing functionality into your own client libraries instead of rewriting it all for such a tool? That's why there's unlikely to ever be such a standard tool.

Share this post


Link to post
Share on other sites
Quote:
Original post by Kylotan

In theory, you could build up something that would allow you to plug in a network protocol and to have some sort of local game representation. But once the user is forced to provide all this, the tool barely gives any additional benefit on top. Why not just build the stress-testing functionality into your own client libraries instead of rewriting it all for such a tool? That's why there's unlikely to ever be such a standard tool.


To simulate users, it's perhaps best to forget a wire protocol.

Once such servers are capable to provide adequate semantics about the world, the one way would be to have alpha testers play the game while everything is logged at gameplay level.

Then, using some genetic/learning/adaptive algorithm, transfer this behavior into bots. They might be able to discern enough information to provide representative behavior.

A bot for testing FPS will be nothing like an MMORPG or SecondLife bot. But this would give convenient generation of realistic bots.

The advantage is obviously that these tests are either fully deterministic (bot causes an exception - you fix the error but keep that particular bot going as a unit test from then on) or evolving (find new obscure errors). And, they can run as often as needed - alpha testers can't.

Who knows, technologies are advancing. Who can say that in 20 years, gameplay testing will not be automatic part of build process.

After all, just about any MMORPG can be played more effectively by bots than players (gold farmers etc.)

[Edited by - Antheus on February 29, 2008 10:40:11 AM]

Share this post


Link to post
Share on other sites
Quote:
Were these actions that you could repeat over and over again


Yes, all actions were doable over and over.

Typically, we would run these scripts in a context where a new user is registered, logged in, creates an avatar, and then starts doing an activity for 15 minutes, then logs out. Repeat.

Share this post


Link to post
Share on other sites
Quote:
Original post by titanandrews
Quote:
We wrote LUA scripts to exercise different actions


Were these actions that you could repeat over and over again in a loop or did the logic on the server prevent this? I would think to gather statistical data you would need to perform actions several times on each game instance. For actions that you can only do once during the game, I suppose you have to reset the state back to a point where you can repeat the test? How did you handle this?

-B



When I have performed stress tests with external programs / scripts on my game server(s), if there are any conflicting rules in the server I simply just temporarily comment it out or create a boolean (often just a const to prevent the hassle of dealing with more variables in the config files) to control the state. For example, at one point I wanted to test for certain packets crashing the server or causing unexpected results so I just toggled off the bandwidth throttling then made a separate application to send the data resulting in a login then just sent a never-ending series of packets to the server until it crashed. A re-login request was sent every now and then to ensure the user didn't get kicked.

For what you mentioned with the states, I had to do a bit of this when testing quests. I just accepted the quest, did everything to complete it except for the very final step before what I wanted to test (ie return to the quest-giver for the reward), save everything, then reset the server with saving disabled. Testing was just as simple as resetting the server and client, and even that can be made into an automated one-click process.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this