• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
ramdy

MMO Architecture doubts

11 posts in this topic

Hi all.

I am wondering about the data which is not necesarely part of the persistent world but is required per-player on any computer it may play. For example: objects in property and their location (let's say thousands of items), inventary distribution, GUI bars, keyboard map, preferences, etc.
Storing this info in a database seems a waste, imagine database querying load on each player startup. I have thought to use an FTP server for this task so on a player startup it will download his files from FTP, just wondering if this would be the proper way to do it.


Second doubt is about chat, having an mmo architecture where you have several game-servers where clients will be connected and they locally query server for persistent data, how would a player from game serverA chat with player from game serverB (both players being in same world (DBServer))? GameServerA may write to DBServer and GameServerB player query for any message at DBServer? I doesn't see this very efficent. Would be a good idea using another server in architecture: ChatServer? In this situation a client would have 2 concurrent connections: a socket connection open with a GameServer and another socket connection with ChatServer. Again, would this be the proper way to do it?

Thanks in advance,
Jorge R.
0

Share this post


Link to post
Share on other sites
Most object positions and other object information is typically held in the server program's memory, just as it would be for a non MMO game. Then you send that information to the clients when the client needs it. Most data that is persistent can be stored in the database, but that gets loaded up when the server starts, before players are connected.

GUI bars and the like are nothing to do with the server - the server may send some values to the client, but then the client draws the GUI.

Keyboard mappings and other preferences could be stored server side or client side. For player-specific data you could load it from the database when they log on (because you wouldn't store all player data in memory for the thousands of players that are logged out) but one or two database reads is not expensive.

Regarding chat, you could simply broadcast a chat message across all the game servers and only the game server with the relevant recipients on would need to send it to a client. Chat doesn't constitute a lot of data and it doesn't matter if it takes half a second to arrive, so you can just broadcast all relevant chat messages 2 or 3 times a second if you like. An alternative is to have a totally separate chat server, but it's not necessary. But you also need to ask yourself - if you can chat across servers, what else can you do across servers? If there is any other sort of multiple-server action, you're going to need a proper way of sending messages between servers anyway, and chat is just one of those messages.

(By the way, these are usually called "questions", rather than "doubts". A doubt is a feeling, and a question is what you might ask to resolve the doubt. :) )
1

Share this post


Link to post
Share on other sites
[quote name='Kylotan' timestamp='1300065544' post='4785422']
But you also need to ask yourself - if you can chat across servers, what else can you do across servers? If there is any other sort of multiple-server action, you're going to need a proper way of sending messages between servers anyway, and chat is just one of those messages.
[/quote]

I think its quite common, if you look at WoW for example each "realm" consists of several servers (a series of world servers for the major continents, instance servers, etc, etc) and while some chat channels are local to the zone or instance others (such as guild or group channels) cover all servers associated with a realm, a separate chat server really is one of the easier solutions. (If you use a design with distinct zones the only other cross server communication that is needed is for when players move between zones so setting up a broadcasting system for the chat really is overkill when you could just take a stock IRC server and use a sensible naming scheme for the channels)
0

Share this post


Link to post
Share on other sites
I think that, in general, pretty much every attribute should be stored in a database somewhere -- there are a lot of good reasons for this, such as centralization (one-stop-shopping, ease of backups), robustness (databases are time-tested in high-stress environments), scalability (the issue of database scalability is well understood, both vertically (bigger servers) and horizontally (more servers)), and atomic transactions (in other words, either the whole affect goes through, or none of it -- no partially-applied effects).

Databases are a very good solution, but that doesn't necessarily mean that everything in the game is kicking off a bunch of queries or stored procedures, there's still plenty of room in this world to be smart, batch transactions, cache things, etc. Indeed, if you did simply litter queries and procedures wantonly throughout your code, you would probably outgrow your database server much more quickly than necessary.

Chat is a different issue, however, because chat's simply don't need persistence, and don't really fit all that well into the relational model (or rather, it "fits" just fine, but could be done just as well with simpler techniques). There wouldn't really be anything wrong with, say, taking an open-source, p2p messaging client and hooking its protocols into your game. It would be kind of like Nintendo's messaging system on the Wii (their buddy-code system) -- its just email on the back end, you can even email your Wii account messages, IIRC. There's frankly a lot to be said for adopting these time-tested solutions -- they're well understood, well supported, and its easy to hire an admin for (and you don't have to train them up). Even if you integrate these solutions tightly with your game, to the point where no user would suspect that you had done so, you still get all these benefits on the back end. If I were writing an MMO, I would be leveraging anything and everything I could -- databases, email, chat clients... everything.
2

Share this post


Link to post
Share on other sites
[quote] Most object positions and other object information is typically held in the server program's memory, just as it would be for a non MMO game. Then you send that information to the clients when the client needs it[/quote]
I guess you mean a GameServer (not the DB Server), so each time a player connects to a GameServer it puts in memory all these "per-player" data which may be required by client.
What about storing all the "per-player" data in an FTP (locally connected to the GameServers), where on client connection GameServer load it to memory and Client download from FTP and load it as well to memory(so both, Game Server and client, have same thing in memory) avoiding the continuous or on request resend of this data. Any operation made by client is communicated to GameServer which will make modifications to what it has in memory and on client log-out will store again as a file in the FTP server.

[quote]Chat is a different issue, however, because chat's simply don't need persistence...[/quote]
That's why I see the ChatServer a good solution, mostly to free the DB, chat is something global not even related with the game itself and nothing to see with persistence so why using the server for this? It could be used also for global messages like "closing server in 5 mins", allow clients to create their own roomchannels, etc. The only thing is that a client would need 1 connection with the GameServer, 1 extra connection with the ChatServer.

[quote]If there is any other sort of multiple-server action, you're going to need a proper way of sending messages between servers anyway[/quote]
For example, players in different GameServers trading items. Any suggestion/link on how to do it properly?

Thanks all,
Jorge R.
0

Share this post


Link to post
Share on other sites
My personal recommendation is as follows:
1) Store this persistent information in a database.
2) Load this persistent information from the database when the player logs in and send it to the client whenever it changes (never request this information FROM the client; see my PS).
3) Save this persistent information to the database when the player logs out.

And, depending on your server:
4) If you have trouble with server stability, save this information regularly to minimize player's loss of time.
5) If your game server is distributed (IE, you have multiple servers on separate or virtually separate machines which may all need this persistent information on the same player), create another server that interfaces with the database server which your servers will communicate with to access and store information; and have it cache this persistent information in memory while performing saves when waiting for a query

PS: DO NOT HAVE ANY SERVER FUNCTIONS RELY ON CLIENT FUNCTIONS BEYOND THE DEPENDENCY ON THE PLAYER'S INPUT.
Doing so will open you up to some very predictable and highly undetectable bug exploits by the clever reverse-engineer.
0

Share this post


Link to post
Share on other sites
You should study some MMO's in detail and also study basic anti cheat measures.
It is a very stupid idea to just go into making an MMO blindly.
I had to research for a pretty damn long time on every intricate detail of my server architecture.
I also recommend to look at arcemu source code.
It is an open source wow server emulator that contains very high quality code/architecture
0

Share this post


Link to post
Share on other sites
[quote name='Ravyne' timestamp='1300070922' post='4785447']
I think that, in general, pretty much every attribute should be stored in a database somewhere
[/quote]


What does that even mean? Attribute is a to generic word to use this way.

[quote='Ramdy']
[color="#1C2837"][size="2"]What about storing all the "per-player" data in an FTP (locally connected to the GameServers), where on client connection GameServer load it to memory and Client download from FTP and load it as well to memory(so both, Game Server and client, have same thing in memory) avoiding the continuous or on request resend of this data.[/size][/color]
[size="2"][color="#1C2837"][/quote][/color][/size]
[color="#1C2837"][size="2"]I would not to this, the server and client are completely different types of software with different goals. Of course as Ramdy mentions, by sharing objects you can avoid some certain data exchange and let the client to more logic, but more logic in client also increases what can be hacked by modifying the client.[/size][/color]

[size="2"][color="#1C2837"]I designed an MMO architecture a couple of years ago, you can see it here:[/color][/size]
[size="2"][color="#1C2837"][url="http://www.next-gen.cc/"]http://www.next-gen.cc/[/url][/color][/size]

Many people views the database as the heart of their application, designing everything around that, this probably leads to very database intensive applications, since disk access and transactions are slow you probably get a slow application.
I view the database more like a dump to store and retrieve objects states, I likely avoid accessing the database as much as possible.

Here is how I designed my MMO server:
Connections servers: Responsible for keeping clients socket connected and filter any incoming invalid data.
Account Server: Stores the players account information.
Character Server: Stores the players characters state.
Universe Server: Basically a place holder for world servers. (Not implemented yet)
World Servers: Each world server/servers are managing one world/planet, controls world wide events such as day/night cycles, weather etc. (Not implemented yet)
Area Servers: One or more of these are responsible for managing game objects and terrain in a geographically limited area in a world. All area servers managing the same area are running their own distributed database for store of game objects located in the area.

If would want to send a message to all players, I would send the message to the universe server, which sends the message down to the world servers which in turns sends it to the area servers which finally delivers the message to the players.
In my game, players can not send a message to any other player directly, players have to be nearby to talk or have each other in a "friend system", this is choice of game design, I could however, allow free for all chat by letting the universe server/servers keep a list of logged in players and their process ids.

Game objects states:
Unloaded: If a game object has not been interacted with for a long time, the game objects is only stored in a disk based database. In this state the game object only takes up disk space, no RAM.
Loaded: My guess is that 99.99% of game objects will be queried for the 3D model, position and orientation, therefore, when these properties are asked, they are loaded from the disk based database to a memory database.
Active: If a game object is interacted with, they load their state from the disk or memory database to a living process/thread in the server, ready to receive state changes.




0

Share this post


Link to post
Share on other sites
[quote name='ramdy' timestamp='1300098237' post='4785539']
What about storing all the "per-player" data in an FTP (locally connected to the GameServers), where on client connection GameServer load it to memory and Client download from FTP and load it as well to memory(so both, Game Server and client, have same thing in memory) avoiding the continuous or on request resend of this data. Any operation made by client is communicated to GameServer which will make modifications to what it has in memory and on client log-out will store again as a file in the FTP server.
[/quote]

The primitives you're trying to build are "load blob of data given a key" and "save blob of data given a key."
While FTP does kind-of implement that, it's not the most secure or most high-performance implementation of that primitive. And, what's worse, it doesn't solve any multiple-accessor access patterns -- it's not "ACID" in the database sense.
I suggest using a persistent key/value or name/document store (like CouchDB, RIAK, membase or even mnesia) for this use case. Or, if you'll only ever have a few thousand players active at one time, perhaps just a SQL database with a key and a blob for storage. This is going to make it easier to do transactional updates, and crashes won't leave your files in a partially corrupted state.

When it comes to connections between client and server, you have to separate logical connections from physical connections. You could build three connections, for example:
1) Connection to the global sign-in/database/coordinator server -- each player has a connection, this connection tells the player what game zone/server to connect to.
2) Connection to the game zone server -- this tells the player about other players in the same zone, monsters, combat, etc.
3) Connection to a chat server -- client connects to the appropriate chat room for each game zone, as well as guild chat, player->player chat, etc.

Or you could create a single connection between a client and a gateway, and have the gateway split the messages to the appropriate servers upon receipt. This creates only one connection out to the client, and multiplexes data in virtual channels on top of that.

Either system can work fine.
1

Share this post


Link to post
Share on other sites
[quote name='hplus0603' timestamp='1300299066' post='4786656']
Or you could create a single connection between a client and a gateway, and have the gateway split the messages to the appropriate servers upon receipt. This creates only one connection out to the client, and multiplexes data in virtual channels on top of that.

Either system can work fine.
[/quote]
1 gateway handling all connections redistributing messages to "sub-servers". Would this solution be "scallable"? I mean you are limited by 1 gateway server.
0

Share this post


Link to post
Share on other sites
Blobs are not necessarily a good storage system when put into a database; their implementations are often not very well performing, simply because traditional databases aren't designed to have arbitrary sized data. You also end up having to write tools to pack/unpack the blob and in addition you cannot trivially search the blob data for debugging/maintenance purposes.


Rather than worrying about this all now, why don't you just write the server to use the extant arbitrary data storage system? Just dump the data into text files on a filesystem. If you need it backed up, there are tools for that already. If you need it distributing, there are tools for that. ACID is usually pretty easy to implement on the FS (you write to new files and switch the names).

A real-time video delivery environment I worked on did exactly this -- metadata is stored in a replicated database, so it can be queried as normal. The actual blob data is simply a regular linux filesystem. I was quite surprised by this -- I was expecting it to be a custom system. But it turns out that the linux filesystems have had so much optimisation work done on them that they're fast enough. If users request objects that the servers near to them don't have, an internal bittorrent network just transmits the files from datacentre to datacentre. There's a degree of "prepositioning" logic applied on top of this to reduce the "misses" if object popularity can be predicted.

The only "trick" involved is in giving each object in the system a unique name (easy enough), and then partitioning that name up, so that the storage forms a deeper, narrower tree structure. It turns out that the main limitation of filesystems in this use is that searching directories is a linear process and it's better to use several layers of nesting.

This has a number of advantages, particularly if you're storing something as plain text -- your server can now be grepped, or sedded to maintain the data. You can hand edit the files if you need to.

Above all, you've just saved a ton of development time. Put a facade onto this process and later on you can always switch to a DB/blob mechanism or using a data provision service (eg; AppEngine or Amazon Elastic Block Store).
0

Share this post


Link to post
Share on other sites
[quote name='ramdy' timestamp='1309687144' post='4830568']
1 gateway handling all connections redistributing messages to "sub-servers". Would this solution be "scallable"? I mean you are limited by 1 gateway server.
[/quote]


You're not limited to 1 gateway server, for many reasons:

Currently, I'm running an instance of HAProxy that has on the order of 100,000 simultaneous connections, load-balancing TCP streams. The machine can do a lot more than that, too. Shuffling TCP packets back and forth is pretty simple and well optimized in kernels these days.

You can also use a hardware load balancer, which will do the same thing using dedicated hardware. Those boxes are designed to do many millions of simultaneous connections, spreading them across thousands of back-end machines based on various algorithms (random, sticky, port-based, round-robin, header-based, etc).

Finally, you can do DNS load balancing -- a single name can map to more than one IP address, and each client will essentially pick one random out of the list of addresses (or, more likely, pick the first, and the server returns them in random/rotating order).

Both hardware and software load balancers will also typically do availability checking. When a particular server does not respond, no new sessions are forwarded to that server, so if a single server crashes, it won't leave new connections stranded.
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0