Sign in to follow this  
adicted

Is this a viable architecture for a mmog ?

Recommended Posts

adicted    124
Hi all, I have been looking for information about programming a client/server architecture for a mmog, but i have only found bits of information and nothing really concrete. After all this reading i have figured out a possible solution but I'm not really experienced about network programming and possible issues this implementation could suffer. So I ask about your opinion and constructive crits about this implementation, and a little explanation about why it wouldn't work is welcomed, not just 'it wont work sorry!' Here it is: The world is partitioned into squares that will be called 'cells'( original, isn't it? ), they won't be very big so i will need thousands to fill the world. An array of cells will be called 'Zone'( :) ). Each zone is controlled by a server. Cells can be passed to another server if it's needed to balance the network. In the client side we check wich cells are visible and a request of the information available in the visible cells is sent ( players, npcs, etc. ). Information from various servers at the same time will be required if the player is viewing cells from various servers. The client side will have the cell 'topology' so it can check the cell visibility itself but it won't know wich server owns this cell so i think a master server with information about what server owns each cell will be needed. I would really like to skip this master server but i cant figure how the client will know which server owns a specific cell, because cells can be passed from server to server, with some restrictions. Well it's long enough, I hope it's not the dumbest post in the forum story, as I said before I have never tried such a thing. I'm awaiting your comments, Thanks!

Share this post


Link to post
Share on other sites
_winterdyne_    530
The client should not decide which cells are visible. The server should. This stops someone hacking your client and requesting info on the whole game world.

Other than that, you've got a pretty standard spatial partitioning zone model there.

I cover a lot of this in some of the threads I've posted on this forum. Your servers need to be aware of each other. Since they are usually colocated on a fast lan, intra server traffic is not too scary, providing not too much goes on at server boundaries. If you're not too experienced with the network programming, I'd advise working on a single zone principle at the moment, and use your framework for batching events. Later on, add the inter-server traffic.


Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Thanks winterdyne,

I'm going to read your posts after writing this one.
Don't you think that calculate cell visibility in the server will overload it ?
Of course i will not check the player frustum with every single cell in the world, it will be optimized, but i'm a bit afraid about overloading the server, i want to support as many players as possible.

Share this post


Link to post
Share on other sites
Steadtler    220
IMO, its all about the scalablity of your architecture. You want to make it such that adding more servers to the world cluster will improve the performance... I dont think you can have a "massive" mog on a single-server.

Share this post


Link to post
Share on other sites
I_Smell_Tuna    96
You say that 'zones' are made up of several cells, then go on to say that cells can be passed to different servers. It would probably be best to pass the whole zone instead. I also second the scalability comment, you should design your back-end programming so that you can load your software on the computer, connect it to the network, and have it communicate with the other servers and not have to do anything more than that.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Quote:
Original post by Steadtler
IMO, its all about the scalablity of your architecture. You want to make it such that adding more servers to the world cluster will improve the performance... I dont think you can have a "massive" mog on a single-server.


I agree that you can't have a massive game on a single server, but if one single server can handle 500 players instead of 300 then the network needed is much smaller and a big improvement.

Quote:
Original post by I_Smell_Tuna
It would probably be best to pass the whole zone instead.


I think i didn't explained the concept well, each zone is controlled by a server, then passing a zone is like passing the whole server. What would be wise is pass an array of 10x10 cells, for example, instead of single cells.


While writing this post i think i have got a solution for the cell visibility calculation!
Client side will calculate wich cells are visible from the current point of view, then it sends a request to the server to get the data of these cells, the server will only check if this cells are really visible and send the data if it's true. Simple!

Share this post


Link to post
Share on other sites
I_Smell_Tuna    96
Well if your server is going to check to see if the cell is visible anyway why not just have the client send a request for visible cells then have the server reply with a list of visible cells.

Here is a thread with MMO cluster architecture ideas. http://www.gamedev.net/community/forums/topic.asp?topic_id=348831

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
The client has to determine what cells are visible between the thousands that exist in the game world( as i said it's optimized, not checking each single cell against the frustum ), then send the ID of the visible cells to the server,
and the server only verifies the visibility of requested cells, no more. I think it's faster and doesn't allow cheating.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Procedure is like this:

Client gets the visible cells.
Client send the list of visible cells.
Server checks the cells received, no more.
If list is valid, information of the cells is sent.

Share this post


Link to post
Share on other sites
Sijmen    231
Quote:
Original post by Anonymous Poster
The client has to determine what cells are visible between the thousands that exist in the game world( as i said it's optimized, not checking each single cell against the frustum ), then send the ID of the visible cells to the server,
and the server only verifies the visibility of requested cells, no more. I think it's faster and doesn't allow cheating.


The verification part still requires the server to do the calculation. Also, that calculation is only a tini bit of CPU work, comared to everything else that happens.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Quote:
Original post by I_Smell_Tuna
Where is the client getting the list of visible cells from?



Cells have a fixed size ( let's say 50 x 50 meters ), the client has the list of cells of all the world, servers too. Client just checks based on his view frustum the list of cells and pass it to the server, the server has the same list, server checks if it controls that cell, if not, it has to send the request to the server that controls it.
I have read the post you mentioned, i think the system i'm talking about could handle the crowded situations quite well. If a zone is crowded then the neighbour zone(zone=server) will start to control cells from the crowded zone until the network is balanced.




Quote:
Original post by I_Smell_Tuna
The verification part still requires the server to do the calculation. Also, that calculation is only a tini bit of CPU work, comared to everything else that happens.



Yes it has to do some calculations, but it only has to check a few cells not check what cells are visible from the entire world. Don't you see it?
If the client does the visibility check and the server just sends the information requested it will be faster, of course, but it will allow cheating as said previously.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
I have made three little drawings to understand it better.


In this drawing we see a world partitioned into cells, the black small squares are the cells, this world is 8x6 cells, in a real online world you will have thousands of cells.
The colored parts are zones, the cells in a zone are controlled by one single server, then each zone equals to one server. In the drawing are three zones/servers.
www.3dusers.com/basic.JPG

When a zone is crowded or lagged it sends a help request to the neighbour zones, and the crowded zone passes a few cells to the neighbours
www.3dusers.com/crowded.JPG


Now we see a client and his viewing frustrum.
He checks all the cells in the world and determines that cell 1,2,3 and red are visibles, then it sends "gimme all the stuff that is in cell 1,2,3,red" to the red server/zone because it is the one he currently is.
Red server checks that cell 1,2,3,red are really visible to the client, then it sends the red cell "stuff" to the client because it is the one it owns. For cells 1,2,3 the server sends a "send information in cell X to player X" to the apropiate server.
www.3dusers.com/player.JPG

Share this post


Link to post
Share on other sites
Erim    132
While reading your thread I couldn't help to notice that you are talking about network traffic in combination with view frustum. It's important to note that none of the network traffic should rely on the view frustum. The client will need to know everything that is going on within the cell, not only what's on the screen. I agree with _winterdyne_, the server should handle all that, otherwise it would be way to easy to cheat and overload the server. Checking the player position server-side isn't a performance problem anyway.
Anyhow, good luck with your project!

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
mmm well the client has to get the information of the currently visible cells, and draw the stuff that is going on. To determine the visible cells a frustum culling is done. Am i missing something ?

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
For the cheating problem i have come with the solution of determine the visibility of the cells in the client and then validate these cells in the server, so the server only needs to check a few cells against the client frustum.

Share this post


Link to post
Share on other sites
Steadtler    220
Quote:
Original post by Anonymous Poster
Quote:
Original post by Steadtler
IMO, its all about the scalablity of your architecture. You want to make it such that adding more servers to the world cluster will improve the performance... I dont think you can have a "massive" mog on a single-server.


I agree that you can't have a massive game on a single server, but if one single server can handle 500 players instead of 300 then the network needed is much smaller and a big improvement.



Depends what is your primary concern. Lets say that, for a fixed users-servers performance, your main concern is the maximum number of players that can connect to the cluster without the connection dropping below that performance. Then, scalability becomes the main concern, almost the only concern. Even if each server can only support 30 players, if you have perfect scalability, you can support 10'000's of players by adding enough servers. I dont think its the cost of additionnal servers that prevent WoW or EQ2 clusters from supporting more than a few thbousand players. Its the scalability of their architecture.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Yes, another problem is that i have not the budget of sony online entertainment and if i can tweak a few lines of code and make the server support a few more players without sacrifying scalability then better.
Sure SOE can put thousands of servers online.
If someday i put a world online I don't think i could get more than 2 servers online...

And well, that is what I am questioning from the first post, do you think it could be ( besides of implementation quality )scalable, efficient, etc...? Why/why not ?

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
On a side note I have readed that the reason why the big folks (soe and such) don't get more people in the clusters is for contents production problems.
If bigger quantities of players are in a cluster then worlds need to be bigger and bigger worlds mean more content so they just repeat the world with different clusters.
This is something I have readed but don't know if it's true.
Also, I dont want this thread goes off topic!

Share this post


Link to post
Share on other sites
Steadtler    220
Sorry I didnt want to go offtopic, but just to remind you to keep scalability in mind. Some views:

Quote:
Original post by adicted


The world is partitioned into squares that will be called 'cells'( original, isn't it? ), they won't be very big so i will need thousands to fill the world.



What happens if a lot of your players decides to go in the same cell at the same time? If you cells are not very small, that will happen sooner than later. If your cells *are* very small, then it wont be performant because you will have too many cells. Its a classic problem I think. I see two solutions: Having cells of varying sizes (hard to manage!) or having copies of cells on multiple servers (RAID-like, less efficient).

Quote:
Original post by adicted
so i think a master server with information about what server owns each cell will be needed.
I would really like to skip this master server but i cant figure how the client will know which server owns a specific cell, because cells can be passed from server to server, with some restrictions.


Since this information is small, of fixed change and does not change a lot quickly, you can keep a copy of this information on every server.

Hope I am of help.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Quote:
Original post by Steadtler
What happens if a lot of your players decides to go in the same cell at the same time? If you cells are not very small, that will happen sooner than later. If your cells *are* very small, then it wont be performant because you will have too many cells. Its a classic problem I think. I see two solutions: Having cells of varying sizes (hard to manage!) or having copies of cells on multiple servers (RAID-like, less efficient).



That could certainly be a problem if the cell has a lot of information attached, or if a group of adjacent cells have a lot of information in conjunction.
The varying size cell solution sounds like really painful to implement!
Maybe instead of passing cells to other zones in overload conditions, the overloaded cells should be shared with other servers ...
I have not really thought about this deep enough but it should be managed in some way...



Quote:
Original post by Steadtler
Since this information is small, of fixed change and does not change a lot quickly, you can keep a copy of this information on every server.

Hope I am of help.



Yes, I think that's the most reasonable/efficient way of managing that.

Thanks a lot Steadtler.



Share this post


Link to post
Share on other sites
hplus0603    11356
Keeping a single cell on multiple servers means you have to do a network message for same-cell object interaction; that's not great for performance.

We implemented variable-shape cells for There; each "cell" is an arbitrary set of nodes in a modified quadtree (modified to map an entire earth-size sphere, not just a square area). We can re-balance the cells "live" (in real time) but we usually don't need to, becuase after an initial settling pass, load is fairly repeating over time.

Share this post


Link to post
Share on other sites
_winterdyne_    530
True visibility calculations aren't required for a client - besides, you need to send traffic from non-visible cells (for example a conversation behind the player).

Your only concern on the server end is what is 'relevant' to the client - typically this is a simple radius affair. You can use spatial partitioning methods to aid in culling for certain event types - but it's a lot less headache to design your world with an effective PVS system - for each segment defining either a procedurally generate PVS or manually reducing the set for certain layouts.

You CANNOT allow the client under any circumstance to dictate to the server what information it should receive. This is asking for trouble. Say you have this layout:

a1 a2 a3 a4 a5 a6
b1 b2 b3 b4 b5 b6
c1 c2 c3 c4 c5 c6
d1 d2 d3 d4 d5 d6
e1 e2 e3 e4 e5 e6
f1 f2 f3 f4 f5 f6

A client has a visual range of 1 cell radius and is in b5. It SHOULD receieve a4-6, b4-6, and c4-6.
Your client could request f1 d2 a1 and anything else it feels like.
You still need to perform the checks on the server. You might as well leave them on the server.

With a regular cell based map, your checks are really trivial - you simply take the cells at the appropriate coordinates. It gets a bit trickier when you have an irregular, hierarchical, structure like I do - which is why I have all that bizarre neighbourhood stuff going on.

If you want to transfer cells from server domain to server domain (don't use the term zone for the area a server handles if it can be discontinuous) you will need a means of distributing that information to all other servers in the cluster - a master server is the simplest way of doing this and handling messages that overflow zone boundaries.

Steadtler is right to encourage you to think about multi-server scaling - sure you can start your game on one server, and possibly get the content together to support say 300 users (still quite a big game!). When those 300 are paying you subs, and their mates are trying to join, you'll want to be able to expand the game with minimal down-time. If you have to rewrite your entire codebase to do so, you run the risk of losing customers. When you're paying a monthly fee for colocated hosting, losing customers sucks.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Quote:
Original post by hplus0603
Keeping a single cell on multiple servers means you have to do a network message for same-cell object interaction; that's not great for performance.

We implemented variable-shape cells for There; each "cell" is an arbitrary set of nodes in a modified quadtree (modified to map an entire earth-size sphere, not just a square area). We can re-balance the cells "live" (in real time) but we usually don't need to, becuase after an initial settling pass, load is fairly repeating over time.



Then your cells are a group of nodes of a quadtree and you split the cell if needed isn't it? It's almost like "Steadtler's varying size cell".
Then a solution when a cell is overloaded would be split the cell in 2 and send one part to another server.
I need to think the pro and cons of that.

thanks hplus0603.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Quote:
Original post by _winterdyne_
True visibility calculations aren't...



Thanks for all the info winterdyne.

But for example positions and states of players that are in "radius range" but not visible shouldn't be sent by the server because it's not relevant, isn't it ?
Maybe I can have two sets of data, one that is radius dependent and another that is view dependent ¿? or will it be too much work for little performance improvement?

I have not said i don't need scalability, in fact it's a must.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this