Jump to content

  • Log In with Google      Sign In   
  • Create Account

Banner advertising on our site currently available from just $5!


1. Learn about the promo. 2. Sign up for GDNet+. 3. Set up your advert!


MMO Architecture doubts


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
11 replies to this topic

#1 ramdy   Members   -  Reputation: 141

Like
0Likes
Like

Posted 13 March 2011 - 03:37 PM

Hi all.

I am wondering about the data which is not necesarely part of the persistent world but is required per-player on any computer it may play. For example: objects in property and their location (let's say thousands of items), inventary distribution, GUI bars, keyboard map, preferences, etc.
Storing this info in a database seems a waste, imagine database querying load on each player startup. I have thought to use an FTP server for this task so on a player startup it will download his files from FTP, just wondering if this would be the proper way to do it.


Second doubt is about chat, having an mmo architecture where you have several game-servers where clients will be connected and they locally query server for persistent data, how would a player from game serverA chat with player from game serverB (both players being in same world (DBServer))? GameServerA may write to DBServer and GameServerB player query for any message at DBServer? I doesn't see this very efficent. Would be a good idea using another server in architecture: ChatServer? In this situation a client would have 2 concurrent connections: a socket connection open with a GameServer and another socket connection with ChatServer. Again, would this be the proper way to do it?

Thanks in advance,
Jorge R.

Sponsor:

#2 Kylotan   Moderators   -  Reputation: 3346

Like
1Likes
Like

Posted 13 March 2011 - 07:19 PM

Most object positions and other object information is typically held in the server program's memory, just as it would be for a non MMO game. Then you send that information to the clients when the client needs it. Most data that is persistent can be stored in the database, but that gets loaded up when the server starts, before players are connected.

GUI bars and the like are nothing to do with the server - the server may send some values to the client, but then the client draws the GUI.

Keyboard mappings and other preferences could be stored server side or client side. For player-specific data you could load it from the database when they log on (because you wouldn't store all player data in memory for the thousands of players that are logged out) but one or two database reads is not expensive.

Regarding chat, you could simply broadcast a chat message across all the game servers and only the game server with the relevant recipients on would need to send it to a client. Chat doesn't constitute a lot of data and it doesn't matter if it takes half a second to arrive, so you can just broadcast all relevant chat messages 2 or 3 times a second if you like. An alternative is to have a totally separate chat server, but it's not necessary. But you also need to ask yourself - if you can chat across servers, what else can you do across servers? If there is any other sort of multiple-server action, you're going to need a proper way of sending messages between servers anyway, and chat is just one of those messages.

(By the way, these are usually called "questions", rather than "doubts". A doubt is a feeling, and a question is what you might ask to resolve the doubt. :) )

#3 SimonForsman   Crossbones+   -  Reputation: 6656

Like
0Likes
Like

Posted 13 March 2011 - 08:37 PM

But you also need to ask yourself - if you can chat across servers, what else can you do across servers? If there is any other sort of multiple-server action, you're going to need a proper way of sending messages between servers anyway, and chat is just one of those messages.


I think its quite common, if you look at WoW for example each "realm" consists of several servers (a series of world servers for the major continents, instance servers, etc, etc) and while some chat channels are local to the zone or instance others (such as guild or group channels) cover all servers associated with a realm, a separate chat server really is one of the easier solutions. (If you use a design with distinct zones the only other cross server communication that is needed is for when players move between zones so setting up a broadcasting system for the chat really is overkill when you could just take a stock IRC server and use a sensible naming scheme for the channels)
I don't suffer from insanity, I'm enjoying every minute of it.
The voices in my head may not be real, but they have some good ideas!

#4 Ravyne   GDNet+   -  Reputation: 10336

Like
2Likes
Like

Posted 13 March 2011 - 08:48 PM

I think that, in general, pretty much every attribute should be stored in a database somewhere -- there are a lot of good reasons for this, such as centralization (one-stop-shopping, ease of backups), robustness (databases are time-tested in high-stress environments), scalability (the issue of database scalability is well understood, both vertically (bigger servers) and horizontally (more servers)), and atomic transactions (in other words, either the whole affect goes through, or none of it -- no partially-applied effects).

Databases are a very good solution, but that doesn't necessarily mean that everything in the game is kicking off a bunch of queries or stored procedures, there's still plenty of room in this world to be smart, batch transactions, cache things, etc. Indeed, if you did simply litter queries and procedures wantonly throughout your code, you would probably outgrow your database server much more quickly than necessary.

Chat is a different issue, however, because chat's simply don't need persistence, and don't really fit all that well into the relational model (or rather, it "fits" just fine, but could be done just as well with simpler techniques). There wouldn't really be anything wrong with, say, taking an open-source, p2p messaging client and hooking its protocols into your game. It would be kind of like Nintendo's messaging system on the Wii (their buddy-code system) -- its just email on the back end, you can even email your Wii account messages, IIRC. There's frankly a lot to be said for adopting these time-tested solutions -- they're well understood, well supported, and its easy to hire an admin for (and you don't have to train them up). Even if you integrate these solutions tightly with your game, to the point where no user would suspect that you had done so, you still get all these benefits on the back end. If I were writing an MMO, I would be leveraging anything and everything I could -- databases, email, chat clients... everything.

throw table_exception("(ノ ゜Д゜)ノ ︵ ┻━┻");


#5 ramdy   Members   -  Reputation: 141

Like
0Likes
Like

Posted 14 March 2011 - 04:23 AM

Most object positions and other object information is typically held in the server program's memory, just as it would be for a non MMO game. Then you send that information to the clients when the client needs it

I guess you mean a GameServer (not the DB Server), so each time a player connects to a GameServer it puts in memory all these "per-player" data which may be required by client.
What about storing all the "per-player" data in an FTP (locally connected to the GameServers), where on client connection GameServer load it to memory and Client download from FTP and load it as well to memory(so both, Game Server and client, have same thing in memory) avoiding the continuous or on request resend of this data. Any operation made by client is communicated to GameServer which will make modifications to what it has in memory and on client log-out will store again as a file in the FTP server.

Chat is a different issue, however, because chat's simply don't need persistence...

That's why I see the ChatServer a good solution, mostly to free the DB, chat is something global not even related with the game itself and nothing to see with persistence so why using the server for this? It could be used also for global messages like "closing server in 5 mins", allow clients to create their own roomchannels, etc. The only thing is that a client would need 1 connection with the GameServer, 1 extra connection with the ChatServer.

If there is any other sort of multiple-server action, you're going to need a proper way of sending messages between servers anyway

For example, players in different GameServers trading items. Any suggestion/link on how to do it properly?

Thanks all,
Jorge R.

#6 nfries88   Members   -  Reputation: 259

Like
0Likes
Like

Posted 15 March 2011 - 05:28 PM

My personal recommendation is as follows:
1) Store this persistent information in a database.
2) Load this persistent information from the database when the player logs in and send it to the client whenever it changes (never request this information FROM the client; see my PS).
3) Save this persistent information to the database when the player logs out.

And, depending on your server:
4) If you have trouble with server stability, save this information regularly to minimize player's loss of time.
5) If your game server is distributed (IE, you have multiple servers on separate or virtually separate machines which may all need this persistent information on the same player), create another server that interfaces with the database server which your servers will communicate with to access and store information; and have it cache this persistent information in memory while performing saves when waiting for a query

PS: DO NOT HAVE ANY SERVER FUNCTIONS RELY ON CLIENT FUNCTIONS BEYOND THE DEPENDENCY ON THE PLAYER'S INPUT.
Doing so will open you up to some very predictable and highly undetectable bug exploits by the clever reverse-engineer.
Looking for paid or open-source C++ programming work. Been programming since 2005. No degree.

#7 wildboar   Members   -  Reputation: 281

Like
0Likes
Like

Posted 15 March 2011 - 06:20 PM

You should study some MMO's in detail and also study basic anti cheat measures.
It is a very stupid idea to just go into making an MMO blindly.
I had to research for a pretty damn long time on every intricate detail of my server architecture.
I also recommend to look at arcemu source code.
It is an open source wow server emulator that contains very high quality code/architecture

#8 flodihn   Members   -  Reputation: 254

Like
0Likes
Like

Posted 16 March 2011 - 03:57 AM

I think that, in general, pretty much every attribute should be stored in a database somewhere



What does that even mean? Attribute is a to generic word to use this way.

What about storing all the "per-player" data in an FTP (locally connected to the GameServers), where on client connection GameServer load it to memory and Client download from FTP and load it as well to memory(so both, Game Server and client, have same thing in memory) avoiding the continuous or on request resend of this data.


I would not to this, the server and client are completely different types of software with different goals. Of course as Ramdy mentions, by sharing objects you can avoid some certain data exchange and let the client to more logic, but more logic in client also increases what can be hacked by modifying the client.

I designed an MMO architecture a couple of years ago, you can see it here:
http://www.next-gen.cc/

Many people views the database as the heart of their application, designing everything around that, this probably leads to very database intensive applications, since disk access and transactions are slow you probably get a slow application.
I view the database more like a dump to store and retrieve objects states, I likely avoid accessing the database as much as possible.

Here is how I designed my MMO server:
Connections servers: Responsible for keeping clients socket connected and filter any incoming invalid data.
Account Server: Stores the players account information.
Character Server: Stores the players characters state.
Universe Server: Basically a place holder for world servers. (Not implemented yet)
World Servers: Each world server/servers are managing one world/planet, controls world wide events such as day/night cycles, weather etc. (Not implemented yet)
Area Servers: One or more of these are responsible for managing game objects and terrain in a geographically limited area in a world. All area servers managing the same area are running their own distributed database for store of game objects located in the area.

If would want to send a message to all players, I would send the message to the universe server, which sends the message down to the world servers which in turns sends it to the area servers which finally delivers the message to the players.
In my game, players can not send a message to any other player directly, players have to be nearby to talk or have each other in a "friend system", this is choice of game design, I could however, allow free for all chat by letting the universe server/servers keep a list of logged in players and their process ids.

Game objects states:
Unloaded: If a game object has not been interacted with for a long time, the game objects is only stored in a disk based database. In this state the game object only takes up disk space, no RAM.
Loaded: My guess is that 99.99% of game objects will be queried for the 3D model, position and orientation, therefore, when these properties are asked, they are loaded from the disk based database to a memory database.
Active: If a game object is interacted with, they load their state from the disk or memory database to a living process/thread in the server, ready to receive state changes.






#9 hplus0603   Moderators   -  Reputation: 6666

Like
1Likes
Like

Posted 16 March 2011 - 12:11 PM

What about storing all the "per-player" data in an FTP (locally connected to the GameServers), where on client connection GameServer load it to memory and Client download from FTP and load it as well to memory(so both, Game Server and client, have same thing in memory) avoiding the continuous or on request resend of this data. Any operation made by client is communicated to GameServer which will make modifications to what it has in memory and on client log-out will store again as a file in the FTP server.


The primitives you're trying to build are "load blob of data given a key" and "save blob of data given a key."
While FTP does kind-of implement that, it's not the most secure or most high-performance implementation of that primitive. And, what's worse, it doesn't solve any multiple-accessor access patterns -- it's not "ACID" in the database sense.
I suggest using a persistent key/value or name/document store (like CouchDB, RIAK, membase or even mnesia) for this use case. Or, if you'll only ever have a few thousand players active at one time, perhaps just a SQL database with a key and a blob for storage. This is going to make it easier to do transactional updates, and crashes won't leave your files in a partially corrupted state.

When it comes to connections between client and server, you have to separate logical connections from physical connections. You could build three connections, for example:
1) Connection to the global sign-in/database/coordinator server -- each player has a connection, this connection tells the player what game zone/server to connect to.
2) Connection to the game zone server -- this tells the player about other players in the same zone, monsters, combat, etc.
3) Connection to a chat server -- client connects to the appropriate chat room for each game zone, as well as guild chat, player->player chat, etc.

Or you could create a single connection between a client and a gateway, and have the gateway split the messages to the appropriate servers upon receipt. This creates only one connection out to the client, and multiplexes data in virtual channels on top of that.

Either system can work fine.
enum Bool { True, False, FileNotFound };

#10 ramdy   Members   -  Reputation: 141

Like
0Likes
Like

Posted 03 July 2011 - 03:59 AM

Or you could create a single connection between a client and a gateway, and have the gateway split the messages to the appropriate servers upon receipt. This creates only one connection out to the client, and multiplexes data in virtual channels on top of that.

Either system can work fine.

1 gateway handling all connections redistributing messages to "sub-servers". Would this solution be "scallable"? I mean you are limited by 1 gateway server.

#11 Katie   Members   -  Reputation: 1434

Like
0Likes
Like

Posted 03 July 2011 - 04:45 AM

Blobs are not necessarily a good storage system when put into a database; their implementations are often not very well performing, simply because traditional databases aren't designed to have arbitrary sized data. You also end up having to write tools to pack/unpack the blob and in addition you cannot trivially search the blob data for debugging/maintenance purposes.


Rather than worrying about this all now, why don't you just write the server to use the extant arbitrary data storage system? Just dump the data into text files on a filesystem. If you need it backed up, there are tools for that already. If you need it distributing, there are tools for that. ACID is usually pretty easy to implement on the FS (you write to new files and switch the names).

A real-time video delivery environment I worked on did exactly this -- metadata is stored in a replicated database, so it can be queried as normal. The actual blob data is simply a regular linux filesystem. I was quite surprised by this -- I was expecting it to be a custom system. But it turns out that the linux filesystems have had so much optimisation work done on them that they're fast enough. If users request objects that the servers near to them don't have, an internal bittorrent network just transmits the files from datacentre to datacentre. There's a degree of "prepositioning" logic applied on top of this to reduce the "misses" if object popularity can be predicted.

The only "trick" involved is in giving each object in the system a unique name (easy enough), and then partitioning that name up, so that the storage forms a deeper, narrower tree structure. It turns out that the main limitation of filesystems in this use is that searching directories is a linear process and it's better to use several layers of nesting.

This has a number of advantages, particularly if you're storing something as plain text -- your server can now be grepped, or sedded to maintain the data. You can hand edit the files if you need to.

Above all, you've just saved a ton of development time. Put a facade onto this process and later on you can always switch to a DB/blob mechanism or using a data provision service (eg; AppEngine or Amazon Elastic Block Store).

#12 hplus0603   Moderators   -  Reputation: 6666

Like
0Likes
Like

Posted 04 July 2011 - 12:52 PM

1 gateway handling all connections redistributing messages to "sub-servers". Would this solution be "scallable"? I mean you are limited by 1 gateway server.



You're not limited to 1 gateway server, for many reasons:

Currently, I'm running an instance of HAProxy that has on the order of 100,000 simultaneous connections, load-balancing TCP streams. The machine can do a lot more than that, too. Shuffling TCP packets back and forth is pretty simple and well optimized in kernels these days.

You can also use a hardware load balancer, which will do the same thing using dedicated hardware. Those boxes are designed to do many millions of simultaneous connections, spreading them across thousands of back-end machines based on various algorithms (random, sticky, port-based, round-robin, header-based, etc).

Finally, you can do DNS load balancing -- a single name can map to more than one IP address, and each client will essentially pick one random out of the list of addresses (or, more likely, pick the first, and the server returns them in random/rotating order).

Both hardware and software load balancers will also typically do availability checking. When a particular server does not respond, no new sessions are forwarded to that server, so if a single server crashes, it won't leave new connections stranded.
enum Bool { True, False, FileNotFound };




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS