In the past days, i've been performing a network stress test, by emulating a server and a lot of clients connecting to it. In the above screenshot, you can see the test window as well as many statistics, which i'll explain in a short while.

The whole server is centralized around two components. The first one is, of course, the network module. I decided to code my own a year ago instead of using an established one ( like Raknet: i heard it didn't support a massive amount of connections, due to its per-packet overhead ). The result is INetwork, an implementation of the RDP protocol ( reliable-UDP ) with a few custom optimizations. It has a low cpu processing overhead, it automatically determines the quality of each connection in the system ( the "ping" ), can concatenate packets together to save header space, or even integrate acknowledgment packets into data packets. It supports many types of reliability, from fully unreliable ( pure UDP packets ) to pure reliable packets ( much like TCP ), with reliable but out-of-order packets in-between. I/O completion ports are experimental under Windows, but i found that the library performance was still excellent without it.

An interesting thing affecting the server performance is the quality of the network card. I already mentionned it in a previous entry, like 6 months ago, but some systems simply do not seem to be capable of handling a large amount of clients; the CPU usage rises extremely quickly, to struggle at around 100 connections. Fortunately, my test computer seems to have a good network card, and i went up to 2500 connections without too much troubles.

The second "core" component in the server is called IClustering. This is a small library which is responsible of determining which entities "see" which others. An entity is determined by a position in space, and a radius. This radius determines the minimum distance at which the entity is being visible by other entities. For example, the entity could be a tree and have a visibility radius of 1 km, which means that any other entity ( like a human ) that is closer than 1 km can "see" the tree. Note that the contrary isn't true: if the human has a visibility radius of 500 meters only, the tree will not "see" him. Each entity can be in 3 states: fully static, semi-static and dynamic. Only semi-static and dynamic entities can be moved inside the space ( or change their visibility radius ). Only dynamic bodies are informed about what they see or not. Technically, the implementation is based off a recursive regular grid, made of NxNxN cells ( each cell being a sub-grid ).

So, with these two components, i "emulated" a MMO server. To do that, i used many concepts:
- zones ( approximately 250 ): entities in different zones cannot see or interact with each other
- places of interest ( the green circles in the screenshot ): these are areas that mostly attract clients. In a Fantasy MMO, these would be cities, buildings or dungeons. In Infinity, they represent planets or stations.
- the clients: the yellow or cyan dots in the screen. They can be in active or idle state.

How it works:
- the server initially creates a number of zones ( 250 in my test ), with a set of places of interest in each one. The places can be recursively formed around other places ( to simulate moons orbiting planets in Infinity, or buildings in cities in other games ).
- a client emulator connects a defined number of clients per second ( between 10 and 100 in my test.. yeah that's a lot of connections/second, but the goal is to stress it after all ).
- each connecting client is affected to a zone. Zones have different probabilities of "interest" ( to simulate zones more busy than other ones ). The most active zones can gather ~30% of all the clients.
- the client is set into active state, and chooses a destination point. There are many probabilities for that. Around 30% to choose an existing place of interest. Around 50% to warp into another zone. Around 20% to choose a location where an existing client is. These numbers are for Infinity, where warping into different systems happens often.
- when a client arrives at destination, he is set into the "idle" mode. If he is warping into another zone, this zone is chosen based on zone probabilities, and is moved to a random point in that zone.
- when a client is in idle state, he has a 0.5% probability to be awaken and to choose a new destination point.

Network-related:
- there are normally 2 "update" packets per second ( to simulate sending position + orientation ), which are around 200 bytes each ( this is randomized a bit too ). These are sent in unreliable mode. They are sent to all the clients an entity ( another client ) is seen by.
- if the client is in idle mode, there are no updates.
- if the client is not seen by anybody, the update rate is lowered by a factor of 4 ( so one update every 2 seconds ).
- two updates per second can look low, but that's what Guildwars is using. And do not forget for Infinity, that ships cannot change of velocity quickly. This should be more than enough even during fights, thanks for dead-reckoning.
- when a client starts or stops to see another entity, a "creation" or destruction packet is sent in reliable mode ( ~300 bytes ).
- the clustering visibility computations are updated 10 times per second

Results:
- they are currently a bit biased, because i have a dual core machine, and the client emulator takes away a lot of resources from the server.
- some zones get a high amount of clients ( up to 150/zone ), most zones get a low amount of clients ( 0-5 ), the average being around 20 clients.
- the visibility radius of the clients is very small compared to the zone and places size. In consequence, most of time there are only packets sent when some clients are in the same place.
- for 1000 clients, the upload bandwidth is around 100 KB, and the download bandwidth around 200 KB. In average, 40% of the clients are idle. The average amount of bandwidth sent by a client to the server is 400 bytes, and the server returns ~500 to 1500 bytes. When many clients (10-15) are in the same place, this can rise to 5000-8000 bytes.

This post is becoming long, i'll add more informations/results later.