Sign in to follow this  

A better way to communicate with relay server?

This topic is 735 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

There's two tasks you want simplified - structs serializing and network communication.

For serialization you might want to check google's Protocol Buffers.

For network stuff there's various solutions, covering different use cases. One of very simple libs is ZeroMQ.

Edited by vstrakh

Share this post


Link to post
Share on other sites
I'd like to understand more what the requirements of your "relay server" are, because I've seen that term used for different things.

What is it relaying from, and to?
What is the rate of communication?
Do you use any kind of authentication?
What kind of session management do you need?
Are there hard latency requirements?

Share this post


Link to post
Share on other sites

Thanks for the reply.

 

It's basically a bridge between game server and oracle database. 

Very often. Possibly more than 50 request per second.

I'm using blowfish encryption with hardcoded session keys.

Session management?

I would very much prefer the fastest communication possible. 

Share this post


Link to post
Share on other sites
That sounds more like an application server to me. Where do the requests to the database come from? Are there game clients that connect to this application server, and the server then issues database requests? Or are requests "stateless" in the sense of a web server -- new requests come in, and should be processed, without persistent connections?

When you say 50 requests per second, is that per user, or overall across all users? How many users at the same time?

Encryption with keys doesn't add security, because someone who has the client software will be able to extract the keys, and if that's all the security you have, they could then spoof any user they wanted.

Session management has to do with how users/clients authorize to the system, and how that authorization stays alive.

"The fastest communication possible" doesn't make much sense. Of course you shouldn't needlessly slow things down. The question here is more "is it more important that communication is reliable, or is it more important that, even if there is a hiccup on request X, request X+1 is hopefully not delayed."

Share this post


Link to post
Share on other sites

Alright so let's say client wants to use an item. The packet goes to game server and gets unpacked. Then game server creates a statement like "opcode, user_id, item_id" and sends it to relay server. Then relay server updates sql procedure with those values and runs the query. Sends a result to game server depending on the query result. Then game server sends its response to client. I'm not very conserned with security between game server and relay server as relay server port will be closed to remote connections.

Share this post


Link to post
Share on other sites
Who hosts the game servers? Do they run on machines that you control? If so, you don't need the relay server; you could just talk directly to the database from the game server machine. (Although a database proxy of some sort might still be useful for other reasons, such as failover or query introspection or connection pooling or whatnot.)

If users control the game servers (there is a "host game" option in the UI) then there really is no difference between a game "client" and a game "server" from the point of view of the database -- either is equally un-trust-worthy, and you need to somehow validate that the request is legitimate, before you turn it into SQL and talk to the database.

Share this post


Link to post
Share on other sites

I'll be hosting the game servers. The reason I'd like to use a relay server is because game server already has heavy work so I'd like to keep database related and maybe some of the sessional client data on relay server and separate them into different machines in future if client amount demands it.

Share this post


Link to post
Share on other sites
I've seen before, an architecture that puts game simulation on one class of hosts (call them "simulation servers"), and game "services" on another class of hosts (call them "application servers.") That can work. (This is a variant of what's called a "multi-tier architecture" -- if you include the game clients for presentation, you end up with four tiers here: presentation, simulation, services, and persistence.) You still should probably think of them as application servers, rather than as "relay servers." A "relay server" is either a network topology router (which is not what you're building here) or it's a forwarding proxy (which is also not what you're building here.)

Also, think a bit about the API to your application server. How will the simulation server express its requests? There are a number of RPC methods you could use, all the way from heavy-weight SOAP requests, to simple XDR RPC, and a number of options in between. You might want to choose something on top of HTTP, such that you could re-use the application servers for other systems you'll need to build if you're successful (customer service, marketing, etc.)

Also not that whatever mechanism you choose, should have an asynchronous API (or you should asynchronize it internally in the game server) to avoid blocking the simulation loop.

Which specific library you use (proxify? thrift? protocol buffers? J2EE?) depends on which language/system you use to implement the stack. If you have a different language/stack on the simulation server than on the database server, you gain flexibility, but the integration work becomes even harder, and making sure both sides agree when you use different code bases to implement the same protocol is an additional challenge. Edited by hplus0603

Share this post


Link to post
Share on other sites

Other than windows API and oracle c++ library I don't have any dependencies. I'm using asynchronous IOCP to handle client connections. As far as the current communication between game server and relay(or rather application) server goes it's a simple tcp connection using very simple custom protocol buffers. An example would be:

/*
0x00, 0x01 : Header
0x02, 0x00 : Character name length
0x41, 0x00 : null terminated character name
*/
unsigned char dbRequest[] = { 0x00, 0x01, 0x02, 0x00, 0x41, 0x00 };

/*
0x01 : Query successful
0x05, 0x00, 0x00, 0x00 : Character stat
*/
unsigned char dbReply[] = { 0x01, 0x05, 0x00, 0x00, 0x00 };

/*
0xFF : Query failed
*/
unsigned char dbReply[] = { 0xFF };

 

To me it looks alright for small number of requests but I imagine it would not be enough if the requests were to burst to a number like 100 momentarily.

 

Connection between client-gameserver also uses a similar approach with better structured protocol buffers. Game server is almost never proactive and only responds when there's a client action.

Edited by AcarX

Share this post


Link to post
Share on other sites
"Protocol Buffers" is a specific data format defined by Google.
If you want a generic names for "binary blobs of data that I know how to parse" then perhaps "PDU" is a better name (for "Protocol Data Unit.")

Also:

0x02, 0x00 : Character name length
0x41, 0x00 : null terminated character name


Belt and suspenders is actually not good engineering practice, because you'll end up using more space than necessary, and more importantly, if one part of the code just uses one option (say, null terminated) then the other may be broken without you noticing (length prefix.)
Thus, pick one, or the other, encoding, and religiously enforce it.

To me it looks alright for small number of requests but I imagine it would not be enough if the requests were to burst to a number like 100 momentarily.


In general, you want to use some kind of language to describe packet contents, and generate the code to read and write those packets. That way, you debug the generator once, and then don't have packet binary data encoding bugs after that.
This is the approach taken in most successful tools/libraries, including Google Protocol Buffers, Apache Thrift, and XDR RPC.

Using a custom binary protocol, instead of formulating HTTP requests and using HTTP as your transport, means that your application server is harder to talk to from the rest of the world. Sometimes, that doesn't matter. And sometimes, you'll kick yourself for getting it wrong :-)

Share this post


Link to post
Share on other sites

Game server creates a statement like "opcode, user_id, item_id" and sends it to relay server. Then relay server updates sql procedure with those values and runs the query. Sends a result to game server depending on the query result.
I'm not sure this is a very good idea alltogether. What do you expect to gain from this? You already stated "asynchronous IOCP", so the one thing about SQL which is painful (queries take a long time) doesn't really affect you.

 

Now, instead of building the SQL statement yourself (which should be an almost trivial string concatenation: stored procedure name followed by two or three parameters) and interpreting the status code whenever it comes in asynchronously, you are building a custom packet and interpreting a custom reply with a status code. Which is pretty much the same (give or take a few dozen cycles), except you now have added extra latency on the millisecond scale to your query processing, and basically doubled the number of network packets on the wire.

 

Traffic is neither free, nor unlimited. Sending out a greater number of packets per second inevitably means that (not always, but in general) more packets will be dropped and you will have to pay more money. Even if internal traffic at your datacenter is "free", it isn't really free (you'll pay for it one way or the other). and its bandwidth sure isn't unlimited.

Share this post


Link to post
Share on other sites

"Protocol Buffers" is a specific data format defined by Google.
If you want a generic names for "binary blobs of data that I know how to parse" then perhaps "PDU" is a better name (for "Protocol Data Unit.")

Also:

0x02, 0x00 : Character name length
0x41, 0x00 : null terminated character name

Belt and suspenders is actually not good engineering practice, because you'll end up using more space than necessary, and more importantly, if one part of the code just uses one option (say, null terminated) then the other may be broken without you noticing (length prefix.)
Thus, pick one, or the other, encoding, and religiously enforce it.

To me it looks alright for small number of requests but I imagine it would not be enough if the requests were to burst to a number like 100 momentarily.


In general, you want to use some kind of language to describe packet contents, and generate the code to read and write those packets. That way, you debug the generator once, and then don't have packet binary data encoding bugs after that.
This is the approach taken in most successful tools/libraries, including Google Protocol Buffers, Apache Thrift, and XDR RPC.

Using a custom binary protocol, instead of formulating HTTP requests and using HTTP as your transport, means that your application server is harder to talk to from the rest of the world. Sometimes, that doesn't matter. And sometimes, you'll kick yourself for getting it wrong :-)

 

I have take a small look into protocol buffers and it looks pretty nice. I'll try to switch to that instead of using my custom "PDU" so thanks for the suggestion. Although my issue still exists which is "how" to transfer data inbetween.

 

 

Game server creates a statement like "opcode, user_id, item_id" and sends it to relay server. Then relay server updates sql procedure with those values and runs the query. Sends a result to game server depending on the query result.
I'm not sure this is a very good idea alltogether. What do you expect to gain from this? You already stated "asynchronous IOCP", so the one thing about SQL which is painful (queries take a long time) doesn't really affect you.

 

Now, instead of building the SQL statement yourself (which should be an almost trivial string concatenation: stored procedure name followed by two or three parameters) and interpreting the status code whenever it comes in asynchronously, you are building a custom packet and interpreting a custom reply with a status code. Which is pretty much the same (give or take a few dozen cycles), except you now have added extra latency on the millisecond scale to your query processing, and basically doubled the number of network packets on the wire.

 

Traffic is neither free, nor unlimited. Sending out a greater number of packets per second inevitably means that (not always, but in general) more packets will be dropped and you will have to pay more money. Even if internal traffic at your datacenter is "free", it isn't really free (you'll pay for it one way or the other). and its bandwidth sure isn't unlimited.

 

Thanks for you reply. Game server is a resource heavy application so I was hoping if, in future, a situation arises where separating game server and database server into different machines will decrease machine load considerably I could provide that without any problem.

Edited by AcarX

Share this post


Link to post
Share on other sites
"how" to transfer data between game servers and application servers?
Use sockets. That's the only way. There are libraries that help you -- they use sockets, underneath.
TCP is fine in the data center, especially for requests that will be asynchronous anyway.

The protocol itself should probably be defined as a set of request/reply pairs.
Request might be "get me a login token, user name is <stirng> and user password supplied is <string>"
Response to that might be "either success, with login token <token>, or failure, with message <message>"

Define your protocol as a set of these pairs. Encode the fundamental data of those pairs of request and replies using some encoding (such as protocol buffers.) Send request on TCP connection; when full reply is available, you have the answer. Success!
Later, you could then re-define the mapping between "data structure" and "network bytes" to support things like making a request through HTTP POST and receiving a reply in JSON format.

Share this post


Link to post
Share on other sites

That is something close to what I'm using currently but rather poorly compared to the way you've described. Would a single TCP connection be enough(for let's say 20 requests average per second) or is a pool of multiple connections is necessary?

Share this post


Link to post
Share on other sites

That is something close to what I'm using currently but rather poorly compared to the way you've described. Would a single TCP connection be enough(for let's say 20 requests average per second) or is a pool of multiple connections is necessary?

No need for more than a single connection. One TCP connection will shove as much data over the cable as will 20,000 connections (actually... 20,000 connections would do less because it's a lot more trouble maintaining them, a lot more small datagrams, a lot more interrupts...).

Share this post


Link to post
Share on other sites
Sign in to follow this