Hi, Does anyone know what tools are used in the industry to do load testing for MMO games? I googled "MMO load testing" and it didn't turn up anything useful. I'm sure companies must perform load testing on their servers so they can know how many users they can support. Does anyone have any insight into this side of the industry? many thanks, B

Any reasonable measure of load must come from user activity - therefore I expect 99% of the tools for this are created in-house, and based on their own client.

You might want to collect samples of user activity, enough to generate reasonably accurate statistics.
For instance, record how many movement packets are sent per second, jumps, actions, and of which size.
Then code a bot that generates traffic based on those statistics and connect several of them to the server.
Depending on the quality of samples collected that should give you an idea of the amount of traffic the server can withstand.

Free MMO Beta Stress Test!!!!!

Best way.

Determining required bandwidth is somewhat "trivial". But any kind of stress testing needs to be performed in real world.

Consider this: You determine that your login server can handle 900 requests per minute.

But in real world, your players will be hammering it before it opens. Once it opens, you'll get 1500 requests in a second!

Gameplay testing. You have determined that your zone handles 40 zone transfers per second, and that in zone, it supports 80 players, each seeing other 79.

Great. Good performance. But then, 150 players get into zone, not everyone sees each other. Within next 30 seconds they get together and (150, each sees ~70), and decide to raid a village in next zone. As they move, zone transfer becomes a bottle-neck. Players start lagging, and respond by clicking repeatedly. This increases your input load by 4 times. This causes latency, some players lose connection. Some connect back, require full state synchronization, others are lagging even more, so they keep spamming the server. Next, latency spikes occur, and you need to start performing rollbacks and resyncs. This in turn increases load, and the whole thing spirals into oblivion.

Artificial load determines some values. But ultimately, real player behavior is hard to simulate.

The worst problem when estimating supported loads is the death spiral. This has been studied extensively for HTTP servers. It doesn't matter what server does until it hits 90% load. Then, the choice of algorithm for handling priorities becomes crucial.

One strategy for this is fastest-first. Always serve fastest clients first, and let slower ones wait. It turns out that this is more effective, although intuition would say that fastest clients will generate more traffic, simply because they get more attention.

The final question also becomes: What is "playable". What if your FPS MMO turns into a 1 update per 5 seconds? What if everyone gets a 5 second lag spike every 9 seconds?

How much do you consider tolerable? There's many informal criteria to determine that, but as is the case with bigger engines, playability is determined by actual players. For some games, the audience might tolerate 1 second RTT. In others, it won't accept 150ms RTT. Can this be covered up with interpolation? Or will that lead only to worse experience?


The goal will ultimately not be: "184.7 players" or "over nine-thousand players". Stress test will determine when gameplay experience degrades. Sometimes it might be 55 players, on another occasion 100.

So in the end, server will likely need to limit number of players, either dynamically or through some pre-set criteria, or degrade the fidelity, by reducing player's AOI or reduce number of updates. Or granularity of them.

Quote:Original post by Antheus
Free MMO Beta Stress Test!!!!!
Best way.

While beta is a good place to get the final testing done, I can't imagine not doing most of it in the alpha builds.

When I did a quick test of the scalability of my client I posted it in the IRC channel and turned off the 5 client per IP. One of my friends in NL just kept opening them (flash client) and I had the client set up for unrealistic input and the server was set to full state updates to see how stressful things might be if the server goes live after maintenance or something and gets pounded by tons of users. Inconclusive since I already imagine it could handle 500 users. I've never had much luck with users stress testing things.

Since input is normally never a problem. Just get a separate computer and randomize delays in it's packets (throwing random garbage packets too). And have it do things that might happen in the game. Like randomly have the client move to different locations. Then if you need more testing just send it to people to run. Then you get the benefit of intensive activity for less alpha testers.

This might seem very obvious but you should be taking a lot of statistics. I use a C# console app so I just output server info every half a second then clear the screen and do it again. Things like average sent in and out average latency, min and max latency, total bytes in and out on average for clients. And tons of other data (like number of players). Also logging and such to quickly make graphs for later on. (peak hours or zones).

I have been designing my map editor so it's connected into my live server cluster to monitor things. Can be useful in tracking hot spots graphically. Basically so you can easily see, "hey everyone is collecting near that place and that's why zone 4 kept peaking at 8 PM".

Thanks for the replies guys!

So I guess what I was thinking was some sort of tool that could "simulate" game play for n number of virtual players. Basically you could describe the protocol to this tool as well as define certain other parameters and the tool could automagically run through all sorts of scenarios. Perhaps there would have to be some sort of AI involved in this. From what you guys are saying, such a tool does not exist and pretty much load testing is done with real users in beta testing scenario. Maybe a tool I am thinking of is just inconceivable?

-B

Quote:We wrote LUA scripts to exercise different actions

Were these actions that you could repeat over and over again in a loop or did the logic on the server prevent this? I would think to gather statistical data you would need to perform actions several times on each game instance. For actions that you can only do once during the game, I suppose you have to reset the state back to a point where you can repeat the test? How did you handle this?

-B

Quote:Original post by titanandrews
So I guess what I was thinking was some sort of tool that could "simulate" game play for n number of virtual players. Basically you could describe the protocol to this tool as well as define certain other parameters and the tool could automagically run through all sorts of scenarios. Perhaps there would have to be some sort of AI involved in this. From what you guys are saying, such a tool does not exist and pretty much load testing is done with real users in beta testing scenario.

I think what was actually said is that it's application specific. To get real figures, you need real players. To get passable figures, you'll still need to use your own client or something based on it. Merely knowing the protocol isn't enough. Imagine we're talking about chess. You can easily come up with a protocol to send a move and receive a move. But the subsequent moves you make depend on the state of the chessboard, so you need a local representation of the game, as well. And that can't trivially form part of any standard tool.

Quote:Maybe a tool I am thinking of is just inconceivable?

In theory, you could build up something that would allow you to plug in a network protocol and to have some sort of local game representation. But once the user is forced to provide all this, the tool barely gives any additional benefit on top. Why not just build the stress-testing functionality into your own client libraries instead of rewriting it all for such a tool? That's why there's unlikely to ever be such a standard tool.