Hi community,

here is Emanuele from Crimson Games Development Department.
An user asked me about how I am dealing with main MMOGs problems in Heroes of Asgard, so I prepared an article about this topic.

So today we will discuss about main optimization problems that you can find in a MMO game development.

I will be happy if someone will add his contribute, so we can learn together: I will add it to open post!

DEFINITION

By definition, a MMOG should allow you to play with a huge amount of people at once and interact with them as if you are in a normal multiplayer game, this in a persistent world.
Now, if we want to dissect a little more this statement, we will see that this is impossible without applying various “tricks” behind the scenes.


WE ARE COMPLAINING ABOUT PERFORMANCES

You can definitely understand how when the amount of connected players grows, server performances will be degraded.
Many operations on the server are required to operate on all connected players or a subset of them, on all objects around the world, on all monsters and their AI, etc. All these calculations are executed several times per second: imagine, then, to have to iterate over 200 players, having to iterate over 2,000 players or having to iterate over 20,000 players, frame each frame of your server simulation. For each iteration, I have to send packets, make calculations, change positions, etc. There is, therefore, an exponential growth of the computational load for each new connected player.
As you can well imagine, is a very large amount of work for a single machine, this due to an obvious hardware limitation.
Usually, therefore, there is a maximum threshold of concurrent players simultaneously processed, after which the server itself (the physical machine) can not keep up, creating a negative game experience (lag, unresponsive commands, etc).
You can not accept new connections beyond this threshold until a seat becomes available, in order to not ruin the experience for those who are already connected and playing.
You could then start multiple servers on different machines, so you can host more players, but of course they can not interact with players from other servers.
The division into various “server instance” definitely does not fall within the definition of MMOG, as it does not allow you to interact with all players in a persistent world, but it creates different instances of the same world. It is acceptable, of course: but it isn’t what we want to achieve.

That said, what can we do to “bypass” a little bit this problem? And what did I already do for Heroes of Asgard? What I describe is the result of my experience and, therefore, it is also what I provided for Heroes of Asgard, obviously trying to get the best.


WHAT CAN WE DO?

There are several measures that can be applied to improve the maximum threshold. Yes, improve it: there will always be a maximum threshold beyond which it is difficult to go (by maintaining the same hardware, of course).

YOU ARE THE CODE THAT YOU WRITE

As first thing: write good code, with your brain attached to this task and without unnecessary waste of resources. It may seem obvious and trite, but it is not. Wasting resources is equivalent to worsen server’s available resources.
Wasting bandwidth means exhaust it in no time, every single piece of data that is transmitted has to be carefully selected. If I send an extra byte for each user, when my server hosts 20,000 players, it means sending about additional 20KB for each frame.
Wasting CPU cycles is like shooting myself in the foot: the actions you perform must be kept to a bare minimum, add a single more function call per user may mean adding N additional CPU cycles, which for 20,000 users will be N x 20000 additional CPU cycles.
Waste memory (and therefore to allocate unnecessary resources) is harmful: the allocation requires both additional CPU cycles and memory. And system memory ends.
In managed environments, also leave resources allocated causes garbage collection, which may mean spending huge CPU cycles to free resources, instead of serving the players and simulate the world.
Ultimately, wasting resources in your code will ensure that you will spend more money and more frequently to improve your servers (when your userbase increases), in order to maintain acceptable performance.


FIX YOUR SIMULATION

As you certainly know, the simulation of a virtual world can be executed a certain number of times per second by the server. This means that every second, all entities and systems in the world are “simulated” a certain number of times. The simulation can include AI routines, positions/rotations updates, etc. It allows you to infuse ”life” to your virtual world.
The number of times your simulation is performed is called FPS or Frames Per Second. It is obvious that if the simulation is cumbersome and requires time, our hardware will tend to simulate the world less times in one second. This can lead to a degradation of the simulation.
But consider: does we need a big amount of simulations performed by the server? Does we need to strive our hardware in this manner? Can we, however, improve this?
Yes. For most games with few players in the same map, and a high game speed (see the FPS, with a high number of commands) our world can be simulated 60 times per second (or less, obviously it depends on game type).
For a MMOG a more little amount can be enough, depending on the genre.
There is no need to simulate the world many times per second as possible, since this will change the simulation in a minimal way, wasting more resources than necessary.
In Heroes of Asgard, for example, the world is simulated 20 times per second (at the moment).


DO WE NEED TO KNOW ABOUT THE ENTIRE WORLD?

We said that in an MMOG we must be able to interact with other players and with the surrounding environment and I should be able to do it with anyone in the world at that time. Quite right, of course.
But, from the point of view of a player, do you really need to know what a player is doing on the other side of the map? No, not always. Indeed, in the majority of cases this player isn’t interested to know if another player, as example, is walking or not in another far area. Send an information that can not be displayed on the user’s screen is a waste of resources.
This observation is important, it allows us to implement a big optimization.
How can I inform a particular player only on entities that may interest him?
Why not break the map (or maps) in zones? A simple subdivision is grid one: divide the map in N x M zones, where N and M are greater than or equal to 1. This technique is also known as space partitioning or zones partitioning.
In this way, a player can only receive information on the entities contained in its area, without needing to have knowledge of distant entities. If in my map 8000 entities are uniformly distributed and it is divided into a 4 x 4 grid, the player who is in the [1, 1] zone will have the burden of receiving information only about 500 entities. A great advantage, doesn’t it?
But consider: what if the player is on zone’s borders? It will not see the players in the nearby zones, although they are visible.
We can therefore understand that the player will have to be informed about the entities contained in its zone and in zones immediately contiguous.
The size of the zones allows you to optimize a lot this method, so depending on the size of a map the size of the grid can vary , in order to obtain the best effect. Also the shape of the zones can vary, to better fit to the composition of the map.


LOOK FAR AS THE EYE CAN SEE

As mentioned, zone division already offers a decent level of optimization, allowing us to send information about a single entity to the players who really can benefit from them.
But let us ask ourselves a question: can we identify useless information in our zone division (remember that also include those contiguous, so in a regular grid we have to dealt with 9 zones in the worst case)? Of course we can.
Most likely a player does not affect entities outside of his field of view.
If I can not see an entity, I do not care to trace what it is doing, although it may be in my own zone. Then sending information about that entity is a waste of resources.
How can you determine what your server needs to send to a specific player? The easiest way is to trace, in fact, the field of view. Everything within that radius is what matters to the specific player, entities outside are not necessary to the specific player’s world simulation.
And since we already have a zone subdivision, we can simply iterate over the entities in player’s zones of interest (instead of all entities in the map) to determine who is within our field of view. This concept is also called area of interest or AoI.
So, continuing the example before, let’s iterate on 500 entities instead of 8000, to extrapolate those hypothetical 25 which fall within the visual range and exchange information through the network only with them.
From 8000 to 25, a good result: doesn’t it? And without the user suffers of missing information as it does not see them. Indeed, it will notice less use of resources.
You can further enhance the area of interest, by applying various measures:

• organize various levels of visual rays; the most distant entity will receive updates less frequently
• filter the interesting entities depending on the morphology of the map; if an entity is in our sight, but behind a mountain, I can possibly ignore it. This measure, however, (in my opinion) only makes sense if you already use culling for other things, so you don’t introduce additional calculations to filter few other entities

DISTRIBUTE YOUR COMPUTATION LOAD

We already said that a single machine will still have a certain threshold beyond which, despite all the optimizations made, you will experience performance degradation (and thus a bad gaming experience).
Fine, but then why not take advantage of multiple computers simultaneously?
There are obviously different ways to do it.
For example, in Heroes of Asgard each map that composes the world is hosted on a separate process. This causes each map can be hosted on a different physical machine.
Obviously, however, you can go down even more and accommodate sets of zones on separate processes (so a single map may be divided into several parts and hosted by different servers).


SLICE YOUR PIE

You can also combine global services (such as chat) in different server processes, to give to your player the impression that, even being connected to different maps (so different servers), you can interact with distant players. Furthermore, break those services from the main world is getting an additional gain in performance.

RECYCLE YOUR TOYS

As mentioned, allocate memory costs a good amount of resources. So why not reuse what we already allocated? The use of objects pools is of great importance in the multiplayer development. It allows to shift the burden of allocating costs when it can be faced with no problems, for example during bootstrap of our app server.
A monster is defeated and dies? Well, I put it aside. I can use it again when another monster must be spawned, just recovering from my pool.
Of course it is clear that you have to use a certain criteria in order to choose which objects to keep in memory and which are not. Should I keep in memory a pool of a monsters that spawns once a month? No, it may be useless. Should I keep in memory a pool of objects representing the drop of the currency? Yes, it makes more sense.

USEFUL LINKS
Of course, an important part of this thread is for resources. Articles, papers: each thing you think that can be useful on this topic.

Spatial Partitioning
http://gameprogrammingpatterns.com/spatial-partition.html
Objects Pooling
http://gameprogrammingpatterns.com/object-pool.html
Game loop
http://gafferongames.com/game-physics/fix-your-timestep/

Feel free to add your questions or your contribute!

Best regards,
Emanuele

Interesting write-up, thanks for sharing it. I have few questions thought.

How can you determine what your server needs to send to a specific player? The easiest way is to trace, in fact, the field of view. Everything within that radius is what matters to the specific player, entities outside are not necessary to the specific player’s world simulation. And since we already have a zone subdivision, we can simply iterate over the entities in player’s zones of interest (instead of all entities in the map) to determine who is within our field of view. This concept is also called area of interest or AoI.

Q1: Did you measure how much this optimisation affects cpu and bandwidth usage? I assume that this was done to drop bandwidth usage, but I would imagine that it increases the calculation complexity, but I guess it is cheaper if your game server can still respond on time regarding the game update rate, and should be done like this? I had thought that typical method of implementing this is just to use e.g. circle AOI to detect the zones player should receive updates from and not to fine tune it to loop over the area of AOI any further. But I guess in these cases the size of the zones matters as well?

Q2: Do you use some kind of delta updates? or when does the players receive full information of a new player entering to a zone?

Q3: Do you have one general game loop at the server which is sending (world updates mentioned above) and then separate message transfers which are only meant to individual players? e.g. if I buy item from a shop I receive response for that from the same server and it has nothing to do with the world updates.

For example, in Heroes of Asgard each map that composes the world is hosted on a separate process.

Q4: How do manage the processes when the game server goes live or offline? if your game server is made up of multiple processes (maps) which can be shared between computers, I would suspect that discovery and management of these could be difficult?

Q1: Did you measure how much this optimisation affects cpu and bandwidth usage? I assume that this was done to drop bandwidth usage,
but I would imagine that it increases the calculation complexity,
but I guess it is cheaper if your game server can still respond on time regarding the game update rate,
and should be done like this? I had thought that typical method of implementing this
is just to use e.g. circle AOI to detect the zones player should receive updates from and not to fine tune it to loop over the area of AOI any further. But I guess in these cases the size of the zones matters as well?

Honestly, at the moment I collected no detailed data. This will be the topic of another article, when I complete my analysis tools.
With no data, I can surely say that this optimization affect your bandwidth in a positive way, because you will not send data to useless clients.
But, in parallel, you also optimize your CPU usage.
You increase your calculation complexity to populate your proximity lists (usually I do it on a parallel thread, few times in a second, not related with game update rate), but you also decrease your complexity when you need to perform operations over near entities.
As example, if I move my character, the server will need to broadcast your new position.
If you made no optimizations, your server will send this update to all connected clients (or, atleast, you will calculate this list of near clients when you need to broadcast a packet around).
If you made these optimizations, your server already has a proximity list for a specified entity, so you can easily send your packet around.
A circular AoI is what I use too in my solution, but a nice subzone subdivision can help you to optimize the number of entities you need to iterate on to create proximity lists.

Q2: Do you use some kind of delta updates? or when does the players receive full information of a new player entering to a zone?

Not yet, because I don't send the entire entities' state for each update. I have different packets with a really restricted content for different situations.

But it can be a nice solution depending on how you build your network solution.

When a new player enters in my character's AoI, I will receive a packet with particular information, so my client can correctly instantiate the new player and its features.

Q3: Do you have one general game loop at the server which is sending (world updates mentioned above) and then separate message transfers which are only meant to individual players? e.g. if I buy item from a shop I receive response for that from the same server and it has nothing to do with the world updates.

Yes, I have a single game loop per map. But, also, I have a packet receiver loop on a separate thread (I'm currently using Lidgren).
Basically, the game loop allows all entities to update their state and execute their routines. From these routines, various packets can be sent (depending on what happens in server simulation).

When you send an action, instead, your message will be processed by another manager class (depending on action type, it can be processed by a different server), that can modify your player's state (or can influence other entities' state).

Q4: How do manage the processes when the game server goes live or offline? if your game server is made up of multiple processes (maps) which can be shared between computers, I would suspect that discovery and management of these could be difficult?

Not exactly, it isn't so hard as it can seem. With the help of a node manager, you can instantiate and monitor your processes on a single machine, so you can broadcast the internal state of your node manager to your master server, in this way you're informed about processes on all machines.
I will write about this topic in detail in a different article.