Server Threads and Multiple Servers

Started by
8 comments, last by wwwdev 17 years, 8 months ago
Heyas just asking for some design pointers. Lets say on an MMO we have a client server architecture (obviously) while the client sends and receives messages on its own personal socket there is no reason why we should limit the server to one socket. My opinion would be to design the server to listen in several ports (thus creating a server for each part of the code), one thread per listening socket. So for example we could have a thread that listens to login requests. This also is responsible for sending pings to ensure client is still connected. Other thread is solely responsible for chatting, receives a message from client, polls game for target of message, sends message to target. And lets just add the final thread: Game thread. This one receives and processes game data. Though im not into multithreading im assuming there is a way to say that the game thread has more priority than the login thread or some other mechanism. This would mean that whenever possible the server would process login requests and chat messages but would focus mainly only on the game. This sounds like a good model for me, please feel free to comment on it and if you find any flaws i would definately appreciate the input. However there is something i particularly dont like about it: Its based solely on one server. There is no way to break the game into more than one computer. I would rather take each thread and stick them into separate processes. If you choose to run them in the same computer, fine it would be similar to using thread, else it would enable you to speed up the game since the game thread wouldnt have to pause once in a while to process chat messages. My biggest problem here is comunication... in a threaded version I can simply call objects from each thread no? its the same program so there shouldnt be any problem. In a multi process however this is probably not possible... RPC calls would be the minimum required no? Or i would simply implement a network TCP or UDP connection based on a lan that would communicate within each server (chat server, login server and game server). So does anyone know a way to comunicate data between different processes? I mean something like Java's serialisation where you can simple send an entire object through the connection (i think). Is it even possible in java? Im thinking CORBA and ORB... i think I did something like this in uni. Anyone knows how big companies do their stuff? im pretty sure they divide zones per server (the famous server boundaries). Am im thinking correctly? should I try to implement different separate servers and link them via LAN (or directly if we are using the same computer). thanks for reading and hopefuly some input will come Yours truly K
Pain is Inevitable, Suffering is Optional.Unknown
Advertisement
If you're building a distributed game server platform, you are right: you need to talk between processes in a reasonable way.

The solution you choose depends on your technology. If you're using Windows, you could try DCOM (and find it sucks for games). If you're using UNIX, you can try CORBA or even good-old SunRPC (and find that it, too sucks for games). With Java, you can try its RMI (and find that it sucks for games). With .NET you can try remoting (and guess what you'll find? It sucks for games!). Oh, there's a few others, too, such as ACE, or MPI, or other technologies (which, by and large, suck for games).

An alternative is to speak your game networking protocol between the server processes, too. If you already implement a game packet protocol with authentication, etc, it makes sense to re-use that for your internal communication needs.

A third alternative is to use a service-based architecture technology (which will make your solution very enterprisey!) such as SOAP or XML-RPC or even good-old HTTP. It all depends on what you already know, what your goals are, and how much pain (of what kind) you're willing to put up with.
enum Bool { True, False, FileNotFound };
The fastest way to communicate between processes in Win32 or Unix is through shared memory and mutex's (Win32 mutex (faster, binary semaphore) or semaphore (more general) object). Also see events and conditions (link below).

More info here,
and here.

Communicating through sockets might also be fast enough (but will be slower than shared memory).
Thanks guys!
I appreciate the input.

Shared memory would only work if the processes where being run in the same machine, right?
that would defeat the purpost of being able to host each server process in its own machine.

i guess sockets are the way to go then... for as long as each server machine is housed in a very fast intranet.

Again thanks!
Yours Truly
K
Pain is Inevitable, Suffering is Optional.Unknown
Hey I had a similar idea. I planned on using a system where there are three types of network connection.

Client
Server
Client & Server

Basicly the different games servers would interact with eachother and then also send information to the user.

IE: positioning server, login server, combat server, etc

then you would have the users 'client server'

The users client will send server spacific infomation to the various servers, then the server take the infomation and calculates it and passes the information to the next server involved, and then is passed back to the user.

Think of it like a different brain to handle spacific areas of code.

example:

user presses 'attack1' the client sends a message to the 'geomitry server' to make sure the client is in range, etc. then passes the message to the 'combat server' which calculates how much damage is done, type, makes sure the user has the object, etc, then passes the message to the 'target AI server' which will control how the spacific NPC should react, has the tally of its hit points, etc. then decides if it should run or attack back, etc. Then this infomation is passed back to the user.

The idea here is instead of having one server handle the workload, you have several servers which worktogether as a team. The good side to this is if im just walking around the world, not attacking, etc, Im only sending information to the 'geomitry' or 'pathing' server (whatever you want to call it) so the other server loads aren't even being used. To me this is exciting.

The down side to this is there could be a significant delay in processing the information if it keeps getting passed off between servers. The only real way to see which is best is to try both the standard method and this 'server team' type method and see the difference in speed.
I use a service architecture for all network connections. A service (or subscriber) runs in its own thread (as far as each application is concerned) and simply generates or responds to network events. Applications can implement as many services as they want, without conflicting ports, and as many subscribers as they want (even to the same service). The services provide a message protocol and operate under either TCP or UDP by choice at startup. Internal protocol is the remit of the user.

Each game process maintains a subscribers to a 'master process' as well as to relevant neighbours (which are validated by the master process, of which there is only one in a cluster). Each game process also maintains a client service (to which client applications have a subscriber).

Winterdyne Solutions Ltd is recruiting - this thread for details!
Quote:Original post by _winterdyne_
I use a service architecture for all network connections. A service (or subscriber) runs in its own thread (as far as each application is concerned) and simply generates or responds to network events. Applications can implement as many services as they want, without conflicting ports, and as many subscribers as they want (even to the same service). The services provide a message protocol and operate under either TCP or UDP by choice at startup. Internal protocol is the remit of the user.

Each game process maintains a subscribers to a 'master process' as well as to relevant neighbours (which are validated by the master process, of which there is only one in a cluster). Each game process also maintains a client service (to which client applications have a subscriber).


Heyas _winderdyne_

I never worked with services before. I know I could google it right now but would you mind giving me a simple paragraph explaining just why use a service instead of a standard network message via sockets?
Hm.... even i can see that the best answer to this post is Giyf... Ill try and do that later on if I have the time and Ill update this thread with any relevant info I might find.
Pain is Inevitable, Suffering is Optional.Unknown
For our system we have several 'tiers' of communication, all of which may have differing protocols and encryption schemes. For example, we intend internal cluster traffic to use TCP for sequentiality and external client traffic to use UDP. Both are modelled as services / subscriber pairs and provide precisely the same interface to the rest of the codebase.

A 'service' or 'subscriber' is simply an interface to a particular message protocol (and all its send queues) (effectively its own network engine), providing an event subscriber or distributor for the rest of your code. They also provide maintenance features like 'I've been idle for 30 seconds' messages, automatic reconnect and reauthentication etc. It really is up to you what you put in there.

Winterdyne Solutions Ltd is recruiting - this thread for details!
I'm assuming you would have a total of three threads running on the server, not 3 for each client (*that* would, of course, be madness :))

Quote:Original post by BloodWarrior
Though im not into multithreading im assuming there is a way to say that the game thread has more priority than the login thread or some other mechanism. This would mean that whenever possible the server would process login requests and chat messages but would focus mainly only on the game.


Thread priority is generally not a good-enough solution for this - it's usually too coarse-grained - so *when you hit hte point when you care* you'd do it other, more proprietary ways (do the scheduling between threads yourself, for instance). But it will be a long time before you need to worry about it.

Quote:
However there is something i particularly dont like about it: Its based solely on one server. There is no way to break the game into more than one computer. I would rather take each thread and stick them into separate processes.

If you choose to run them in the same computer, fine it would be similar to using thread, else it would enable you to speed up the game since the game thread wouldnt have to pause once in a while to process chat messages.


It probably won't much speed up the game. Bear in mind that even a single-CPU computer is a parallel processing machine with many mini-cpu's hidden all around the mobo, system, etc, dedicated to doing different tasks (e.g. I/O), so that at a simple level a single machine is best off trying to do several different things simultaneously - and the delay added by doing things on different machines reduces speed considerably. To get a significant benefit, you need to really know what you're doing; you're likely to find that the first, second, and third bottlenecks you hit in the system are independent of whether your server runs on multiple machines, or, worse, are CAUSED by it running on multiple machines.

Quote:
So does anyone know a way to comunicate data between different processes? I mean something like Java's serialisation where you can simple send an entire object through the connection (i think). Is it even possible in java?


Sending objects is always painfully slow, just don't do it :). Send raw low-level data, bits and bytes, and be careful of what you send - but, as noted above, I'd not bother for some time yet, because I doubt you'll *know* or *understand* what you need to send for quite a while yet!

Finally, the service route is a good one, in that it gives you a lot of flexibility in the long-term, so that when you DO start to hit problems with one server and KNOW what you are going to have to split on different servers, it's easy to do any of many different ways - whilst not requiring you to do significant extra work to start off with.

Service architectures also have a lot of potential for performance improvements that are not so easy with other setups, mainly due to their flexibilty and modularity, but that's a much much more advanced topic all to itself :).
Hi, this problem can be approached in various ways. See here http://nonblock.tripod.com/hpaftpd.html

This topic is closed to new replies.

Advertisement