# Networked Game Structure

This topic is 2508 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hello!

I'd like to start by referencing these two threads:

http://www.gamedev.n...hrough-gateway/
http://www.gamedev.n..._1#entry1804154

My situation is I, like the above thread starters, have come to a point where I need to decide which path to go. I am using TCP with the servers (Login, Chat, Zones, etc) all running an IOCP. Of the two methods: a gateway or direct connection to said servers (chat, zone, etc) I must say I'm very fond of the gateway implementation, however I have a concern that if the gateway crashes, so do the clients. Referencing WoW or even EQ, you used to be able to connect and if the login server crashed, you would be fine long as you didn't log out.

So my question is how did they do it? Without exposing too many sub servers, what would be the best solution? I'm having a hard time wrapping my head around how I would connect to the Login server ,verify the account, and then pass the verified client to the "sub server."(a sub-server is only exposed via a port on the Server Machine to a server higher up in the chain) For example ... login -> verified -> [???] -> connection to [Chat/Server List] -> [???] connect to realm selected.

|
V

[Chat / Server List]
|
V

[Realm]: -> [Zone]

From the second link it is stated you cannot redirect a socket that is already active to a new server, so I would need to close the original socket that was used to verify the client, recreate the socket.and then connect to the sub server. Is this the proper way? How would one ensure that the client was actually verified in the first place? *scratches head* I hope this is something simple that I am making an elephant out of.

##### Share on other sites
Referencing WoW or even EQ, you used to be able to connect and if the login server crashed, you would be fine long as you didn't log out.[/quote]
WoW used two different sockets to connect to it's client on two different ports.. Essentially you would send handshake and auth info to a login server, which would process your build info and details, send back a result packet and a realm list. You would then choose which realm to connect to - a new connection would be opened to the realm server and the login one would be dropped immediately after connecting.

In other words, in WoW at least, you would connect on two different sockets. Basically, yeah, I hate how you have to do it too, but it seems creating multiple sockets is the only way. Pseudo code:
 Socket s = new Socket(serverAddress); s.send(AuthRequest); ..wait until recv. if(success) Socket r = new Socket(realmAddress); else Error to base screen 

##### Share on other sites

So my question is how did they do it? Without exposing too many sub servers, what would be the best solution?

Your servers need to be flatter. It could look something like:

 Subscriber/User Database ^ ^ | | Subscriber Login Realm Character Database ^ | ^ | | | | +---+--------+---+ | | | | Realm Login Zone Server | ^ ^ | | | +--------------+----------------+ | Client 

The client connects to the subscriber login server, which uses the subscriber database to verify subscription.
This also returns a list of realms that are available. The client chooses one, and connects to the realm login server, which verifies the login credentials, and shows a list of characters.
The client selects a character, and the realm server tells the client which zone it's currently in.
Client connects to the zone server, giving a character and user reference, which the zone server verifies with the realm server, and then instantiates the character in the zone.
At that point, the client can talk to the zone server for all zone gameplay and chat, the realm login server for guild chat and cross-zone chat, and the subscriber server only for specific-player chat (if you support that -- most systems don't).

You probably have many realms that all share a set of subscribers, and many zones within a single realm. Similarly, you probably use Kerberos-style authentication (signed, datestamped tickets) for the hand-off from subscriber to realm, and realm to zone.

##### Share on other sites
Neglected, thank you for your response I figured something like that would be needed due to the desire to move from a "single gateway" to a secondary sever, to avoid "login server/gateway" crashing and the connected clients dropping as well.

hplus0603, first thank you for the detailed post. You gave me more ideas about how to add/implement things that I have done differently, thank you for that. But my question is not about verifying, but how to route a new socket to said "sub-servers." But correct me if I'm wrong; from your post I've come to the conclusion that I will need to expose at least two/three servers. The login/gateway, the realm server, and the chat(global chat e.g. guild /global channels) server. (could combine the two into one but that's a minor detail for now.)

And to do all of this (again please correct me if I'm wrong) but I will need to close and recreate sockets to connect to said servers. e.g.

First Socket: Connect to login server, verify account. Obtain a Kerberos-style Ticket if the account is valid.
Server sends the ticket to the client verifying the user name/password is valid, and a list of realms.

Second Socket: Client closes first socket, recreates a new one that connects to the selected realm.
Client sends the Kerberos-style ticket for validation.
Server verifies. If valid, returns a list of all the characters.

Client is now connected to [Selected Realm] and now all messages can be routed through the [Selected Realm] server to zones/global chats (guild, whispers, etc), no need to connect to specific zone servers as all messages can be routed to the proper Port a zone server is listening on (assuming all zones of a specfic realm are on one machine and the same machine as the realm server itself).

So if the above is correct, would this mean that the login server exposes PORT addresses for all realms? e.g. Login_Server: mygame.com:7501 Realm_Sever: mygame.com:7502 Specific_Realm: mygame.com:7503-7600 (assuming 97 realms ... yeah I know just an example.) and to move from one to another I'd just recreate a socket and connect to the proper port address (with verification happening obviously) With this method wouldn't exposing the realm ports be a bit dangerous in so much that someone can blast the realms with all types of attacks? I'd like to keep the specific realms unexposed .. but I suppose this means using a gateway such as the "Realm Server" which would then be the target of said attacks if they were to happen anyways?
_

##### Share on other sites

hplus0603, first thank you for the detailed post. You gave me more ideas about how to add/implement things that I have done differently, thank you for that. But my question is not about verifying, but how to route a new socket to said "sub-servers." But correct me if I'm wrong; from your post I've come to the conclusion that I will need to expose at least two/three servers. The login/gateway, the realm server, and the chat(global chat e.g. guild /global channels) server. (could combine the two into one but that's a minor detail for now.)

If you want to use a gateway, then you can build a separate gateway (taking the role of "client" in the diagram) and build a network protocol that adds an additional level of framing that the client uses to talk to the gateway. It really doesn't change the architecture much, except you need to expose fewer physical machines directly to the internet. (If you have full control over the services exposed, you can make that as secure as a system with a gateway)

So, instead of "client" in the diagram, it would say "gateway," and the client would connect to a gateway. I would probably make the gateway dumb, so the system would tell the clients "you need to talk to server 3722" and the clients would then ask the gateway to establish a new connection to 3722. To the client, it would probably look a lot like establishing new socket connections, except the 'address' in question is just "server 1234" in a way that the gateway knows to resolve to a particular host.

Note that in a real data center, you typically have a hardware load balancer of some sort (Juniper, Brocade, F5, whatever) which has the public IP/IPs of your service/s, and it does reverse NAT for connections to your array of servers. In some sense, this serves as a "transparent gateway." You'd typically want to allocate one port on the outside for each service/machine combination on the inside in that case.

##### Share on other sites

[quote name='grazing' timestamp='1321273199' post='4883741']
...
... [/quote]

hplus0603, don't take this the wrong way but I love you. A million thank yous!

I'm sure we've all been at a point like this but I've been stumped while attempting to process this situation in my brain for a few days and couldn't quite wrap my brain around it until now.

Thank you again.

##### Share on other sites
Sorry for reviving old post.
For the "traditional" gateway approach the problem I see is that if gateway falls, all clients fall as well.
If you have a list of gateways would be a matter to let the client know IP/Ports and if losing connection just trying to connect to any other from the list without user even noticing.

So the process would be:
2. connect/validates to gateway
3. client start sending messages to gateway, it will redirect to proper internal "sub-server"
...

An image to ilustrate it better:
[attachment=6895:Dibujo.jpg]

*Note Im calling a gateway what in fact is a "home-made" server with functionality reduced to receive messages from client, parse/validate and resend. Gateways would be the only accesible by Internet and gateways and subservers are in same LAN.
** Gateways and subservers can easily scale.

##### Share on other sites
Why bother with all the relay servers? I don't see what it gains you, aside from a lot of extra hardware to purchase and maintain. After all, there's no point running your "gateways" (relays?) on the same boxes as the "subservers" because that just defeats the purpose of redundancy.

I'm personally more a fan of connecting to a point of authority for login, then reconnecting directly to what you call "subservers" for service (be it gameplay, web shopping, or whatever else). The login system gives you a token which can validate your connection to the actual services, and you're done. The authentication system can easily be scaled horizontally for the capacity and fail-safety required, and you're free to scale the back end services as necessary without any intermediate infrastructure.

In other words, you get a (more or less) maximum ratio of fault tolerance to requisite hardware.

##### Share on other sites
Hi,
I will explain before how I have in mind doing it:

All communications are limited to text, what client-network exchange are just text-strings with a specifiq format to identify the command and params, something like:
command&param1&param2&...
In the case a client want to send a message to a chat room, it would send something like:
"chatmessage&GeneralChatroom&Hello all"

Gateway would receive, decrypt and validate the string and depending of the command identify what server is in charge to process it, so it ends resending the string to the proper server in the network.

*Note what I call gateway is a server with traffic redirection based on message content as main duty.

Benefits of the gateway:

1. One IP per gateway, no gateway means you would need 1 IP per machine in cluster?
2. Frees of some network programming aspects client and "sub-servers", they just handle 1 connection for everything.

The login system gives you a token which can validate your connection to the actual services[/quote]
Yes, login is before all this. client and gateway obtain a token per session which is used also as key for message encryption.

Replace "gateway" by "proxy" and this is almost what I mean:
http://realmcrafter.com/store/pages.php?pageid=12

##### Share on other sites
If you get to the point of engineering some kind of gateway server IP fall-back, then you should take a step back. You're doing it wrong.

Buy a hardware load balancer. In fact, buy two. Link them in "high-availability/hot-standby" mode. If one dies, the other can immediately jump in and take over.
This hardware load balancer is the interface between the internet and all your servers. It also serves as a firewall of sorts. You can use a single IP address on the outside, and map a number of ports (say, any port between 1 and 65535) to a different host and port on the inside.
An alternative is to use the load balancer to do round-robin balancing. It will poll your web servers (or gateways) for up status, and send each new incoming connection to one of the available, up, target hosts.

Yes, you may pay many thousand dollars for this hardware, but when you're at the point where you need gateways and forwarders and whatnot, the leasing cost of this hardware is a very very small part of your budget.

1. 1
2. 2
3. 3
Rutin
15
4. 4
khawk
14
5. 5
frob
12

• 9
• 11
• 11
• 23
• 12
• ### Forum Statistics

• Total Topics
633661
• Total Posts
3013221
×