• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
lerno

Game server DoS / DDoS mitigation strategies?

29 posts in this topic

Are there any must-have DoS / DDoS mitigation strategies one should always build into a server?

 

Regardless of TCP or UDP, it feels like there is very little one could do against a DDoS which tries to saturate the network.

 

Even if the attacker can't do that, simply sending login packets with spoofed IP addresses/ports could be a problem. For something like SSL, it's possible to hit the CPU hard by initiating a handshake. CPU exhaustion attacks can be mitigated by puzzle challenge-style logins, but it should be fairly easy to block login by making sure that the all login "slots" are in use.

 

(I know some websites that use CloudFlare, but that's for serving http/https content and so isn't an option)

 

Any opinions of what a reasonable line of defence is?

0

Share this post


Link to post
Share on other sites

With the mentioning of cloudflare and in AllEightUps post number 4 he uses a ssl connection for initial id/salt I saw recently that the new beta firefall uses simple https requests to aws servers for login. You could do similar use cloudflare to prevent alot of attacks on the initial login atleast. You would still have to worry about the bandwidth issues after logged in though. Depending on what kind of game it is if its not real time you could possibly get away with all http requests using the cloudflare or other technology thats already in place.

0

Share this post


Link to post
Share on other sites

Good ideas all around. Some questions:

 

AllEightUp:

For bandwidth-limiting, you're thinking of a simple thing to prevent the client from issuing requests and getting response from the server for the request, right? Because nothing's going to prevent the packets actually hitting my server. Or are you thinking of some configuration I could do to firewalls?

 

For UDP I have encryption, but only after the UDP payload is spliced into sub-commands. I don't have it on the UDP protocol level. I don't know if that's a problem or not. For TCP the encryption happens after I split the stream into packets, but that feels "safer".

 

jeff8j:

Doing login over https has occurred to me exactly for that reason. Right now it uses a puzzle-challenge and limited evaluation pipe to prevent overload. Still, that won't protect it against bandwidth saturation. On the other hand, say I go with something like aws, then this might even make it worse. After all, killing the login server with requests is fairly harmless. However, if they fail to do so, or are able to get a large amount of login tickets, then they might start targeting the lobby and game servers - and those are the ones I would like to protect.

 

I've made secure login for game/lobby as cheap as possible by relying on a kerberus-style ticket to set up encryption, but they are hitting the db after the security handshake... The nice thing about the login servers are that they don't affect any sub systems if overloaded.

 

It's hard for me to tell if moving the login to a simple https service would help or not.

 

hplus0603:

What would I gain by filtering but my game port? Assuming I have a firewall closing everything else, what would they get by bombarding other ports, as opposed to simply the game port?

0

Share this post


Link to post
Share on other sites

There are also other things you need to be sure to handle, one of the latest and greatest attacks is the simplest of all:

 

A pseudo slow client.

 

The trick is to establish TCP connections, valid ones, but send as little data as possible. Since data is trickling over the connection, the bad connection remains open, consuming a great deal of resources. It is, in effect, the same as a TCP Syn flood attack, except the connections actually complete.

 

There are, of course, the other things you need to handle:

 

Validate everything, never trust data coming from the client.

 


What would I gain by filtering but my game port? Assuming I have a firewall closing everything else, what would they get by bombarding other ports, as opposed to simply the game port?

A software firewall on a port can "help" to a degree, but the machine still ends up having to handle the data getting sent to it. While the data is simply discarded, it is still being sent to the machine.

 

Hardware firewalls are a bit better on this, since they're actually designed for throughput and packet streaming, so they can tend to discard blocked traffic at a much faster rate, but again... that's still traffic ON THE WIRE. If you're getting charged for bandwidth, you'll get charged regardless of if you accepted the traffic or not.


I've made secure login for game/lobby as cheap as possible by relying on a kerberus-style ticket to set up encryption, but they are hitting the db after the security handshake... The nice thing about the login servers are that they don't affect any sub systems if overloaded.

 

Who is hitting the database? The client should never touch the database. In fact, the service the client talks to should, ideally, not talk directly to the database. The database should be firewalled, locked away behind services that do all the intermediate actions. Those services should be located on servers that are locked away from the rest of your machines, if at all possible. The simple fact is, a server on the internet is a gateway. Once someone gets into your gateway, then you need to limit their lateral movement. A database server sitting on the same network as said gateway is a ripe target to get its data ripped. But a database server locked behind some service gateways and  firewalls is a much harder target to access and would give you time to notice and stop said infiltration.

0

Share this post


Link to post
Share on other sites

Washu: In regards to the port blocking, say I have the game port at 6000. What does an attacker gain by flooding 6001 that the attacker can't gain by flooding 6000? That's what I was wondering.

 

In regards to the db - it's the server that connects to the db of course, I was referring to the fact that a login / request by the client may cause the server to perform a db query.

 

In the case of a DoS attack on a server, then that server will trigger many db requests. If the queries are good and the server has a connection pool that limits the number of simultaneous queries to the db, that shouldn't be a problem. Still, I prefer to avoid enabling a player to cause a db query as far as possible.

 

Like you said, the db should never be accessible directly from the internet, and preferably thoroughly locked away inside the internal network.

 

In my case I was imagining the setup looking a bit like this:

 

 

 
              Internet
                  |
                  |
                  V 
            +----------+
            | Firewall |  some.domainnamehere.net <- Firewall connected to internet
            +----------+     
      6000    |  |  |    6200               Port forwarding to the right server
   ___________|  |  |____________
   |             |              |
   |             | 6100         |
   |             |              |
   v             v              v
+-------+    +-------+      +------+
| Login |    | Lobby |      | Game |   <- Servers not directly connected to net
+-------+    +-------+      +------+
                 |              |
                 |_____    _____|
                      |    |
                      v    v
                    +--------+
                    |   Db   |
                    +--------+
Edited by lerno
0

Share this post


Link to post
Share on other sites

Washu: In regards to the port blocking, say I have the game port at 6000. What does an attacker gain by flooding 6001 that the attacker can't gain by flooding 6000? That's what I was wondering.

Not much, other than increasing your traffic bill. They're much more likely to attempt to flood an open port than one that's closed. Mainly because the goal is to swamp the machine, and the easiest way to swamp a machine is to use up the kernel's free memory by clogging it up with useless garbage through open ports.
 

In regards to the db - it's the server that connects to the db of course, I was referring to the fact that a login / request by the client may cause the server to perform a db query.
 
In the case of a DoS attack on a server, then that server will trigger many db requests. If the queries are good and the server has a connection pool that limits the number of simultaneous queries to the db, that shouldn't be a problem. Still, I prefer to avoid enabling a player to cause a db query as far as possible.
 
Like you said, the db should never be accessible directly from the internet, and preferably thoroughly locked away inside the internal network.
 
In my case I was imagining the setup looking a bit like this:

Lobby and Game appear to be on the net to me. If you're forwarding to them, they're on the net.

The typical behavior is to have a proxy server that sits between the servers that do the game logic and the player. The proxy server does all the validation, load balancing, packet validation, etc. Then it forwards the player requests on to the game logic servers. Those servers do the processing of things like physics, making sure that the actions are acceptable, and then issue a response to the proxy server that tells it aye or nay to the player's request, along with publishing relevant state information.

There would be a DMZ for the proxy servers, and then an internal network for the game logic servers.
1

Share this post


Link to post
Share on other sites

AllEightUp:

For bandwidth-limiting, you're thinking of a simple thing to prevent the client from issuing requests and getting response from the server for the request, right? Because nothing's going to prevent the packets actually hitting my server. Or are you thinking of some configuration I could do to firewalls?

 

For UDP I have encryption, but only after the UDP payload is spliced into sub-commands. I don't have it on the UDP protocol level. I don't know if that's a problem or not. For TCP the encryption happens after I split the stream into packets, but that feels "safer".

 

The bandwidth limiting is basically a preventative measure.  A common attack vector is for a client to login correctly and everything looks nice but they start sending valid commands as fast as possible.  For instance, say you have movement messages, instead of sending a movement message packed in with other data at say 10 times a second, they start sending just the movement update alone at 1000 times a second.  Everything is 'valid' but they are hitting you in two places, the bandwidth of course and the CPU overhead of dealing with the valid but excessive packets.  When detected, you boot the connection and stop accepting the packets, that brings CPU/memory on the server back to normal even if you are still getting slammed with the excessive bandwidth.  If they keep it up even if you keep booting and ignoring them, normal mitigation such as telling the router to bounce the packets for an hour and of course contacting the source ISP or backbone provider and informing them of the ip source etc.

 

As to the encryption, I don't generally trust myself to get all the edge cases correct and rather use the separate SSL connection for initialization.  The ssl connection is only used for login and other secure items.  As a benefit, you can use it for the initial NAT traversal since you have the connection anyway.  In general though, this tcp connection is only there for a brief period and won't interfere with the UDP once initiated.  This is mostly paranoia on my part, I just don't want to risk that I goof something up which allows someone to crack the UDP encryption somehow and grab up everyone's passwords.  It's not paranoid if everyone really is out to get ya... :)

1

Share this post


Link to post
Share on other sites

Lerno:

Using https wouldnt protect the game servers directly but could do woners as far as login and lobby systems. With the pro cloudflare you can set the cache to 30 seconds and they have servers all over the world so when people hi the lobby they can bring up the cached version 30 seconds isnt that much of a difference then the cloudflare servers would only touch your server once every 30 seconds. You would probably want to whitelist the cloudflare servers and block anything else through a firewall. Also the nameservers would point to cloudflare not your ip so you can keep your ip for login/lobby not known to the public. You could also have a fall back server or multiple if some get in trouble. You would also have the servers report over https to keep that login/lobby server masked. Overhead would also drop as you dont have to serve https from your server you could serve http and have cloudflare do all the heavy https work.

 

Note: If your game could use websockets you could block ips and cloudflares ddos would still work to prevent them from even touching your servers. This does have the downside of its not a direct connection so latency will go up a bit but I have done this and its not that bad I figure because cloudflare trys to be as close to the end user as possible so its pretty much in line on the way to the server.

 

In the end the only way to be slightly confident that things wont go down is to have enough servers in enough locations with big pipes that it would be very difficult to bring them down and if one goes down another handles the load. Thats how google, facebook and the like handle things and thats why a web option is easier to get going something like cloudflare has many powerful servers in place around the world with massive pipes I know I couldnt afford.

 

Of course there are alternatives to cloudflare some way better but none at the same level of cost $20 a month or even free for basic protection so thats why I keep going back to it.

0

Share this post


Link to post
Share on other sites


Lobby and Game appear to be on the net to me. If you're forwarding to them, they're on the net.

The typical behavior is to have a proxy server that sits between the servers that do the game logic and the player. The proxy server does all the validation, load balancing, packet validation, etc. Then it forwards the player requests on to the game logic servers. Those servers do the processing of things like physics, making sure that the actions are acceptable, and then issue a response to the proxy server that tells it aye or nay to the player's request, along with publishing relevant state information.

That sounds like a lot of work unless validation/packet validation is heavyweight - and that should depend a lot on the type of game. I personally worked on poker server backends for a few years and I never ran into a need to do something like that.

 

That said, the same company worked on a "next gen" poker platform which distributed everything, even worse than what you describe. Scalability and reliability was in the dust for that platform.

 

So to me that solution sounds like quite the trade-off.

 

I'm also wondering what sort of attack you'd do on a game server with a single open port. Yes, it's possible to overload it, but killing the proxy by overloading it would amount to the same thing, wouldn't it? Please explain the benefits, because I don't really see them.

0

Share this post


Link to post
Share on other sites


As to the encryption, I don't generally trust myself to get all the edge cases correct and rather use the separate SSL connection for initialization.  The ssl connection is only used for login and other secure items.  As a benefit, you can use it for the initial NAT traversal since you have the connection anyway.  In general though, this tcp connection is only there for a brief period and won't interfere with the UDP once initiated.

 

That's good advice in general. I've dug deep into cryptography to design a protocol which I feel fairly confident in. Mostly because it's basically an implementation combining two well known protocols. Still, I know it's a risk.

 

(Even if it's broken, I don't actually use any user generated passwords. In addition, gameplay is mostly anonymized. It's not really possible to determine the e-mail of another player, and since game is partitioned into separate worlds, if you get hold of someone's credentials it's unlikely to be someone playing in the same world as you.)

0

Share this post


Link to post
Share on other sites


Using https wouldnt protect the game servers directly but could do woners as far as login and lobby systems. With the pro cloudflare you can set the cache to 30 seconds and they have servers all over the world so when people hi the lobby they can bring up the cached version 30 seconds isnt that much of a difference then the cloudflare servers would only touch your server once every 30 seconds. You would probably want to whitelist the cloudflare servers and block anything else through a firewall. Also the nameservers would point to cloudflare not your ip so you can keep your ip for login/lobby not known to the public. You could also have a fall back server or multiple if some get in trouble. You would also have the servers report over https to keep that login/lobby server masked. Overhead would also drop as you dont have to serve https from your server you could serve http and have cloudflare do all the heavy https work.

 

In my case all of login/lobby/game are plain sockets. No http. That said, it would be possible to convert them to http(s) services. However, cached data would not do. Both login and lobby servers serve personalized results, and for security reasons the login server shouldn't serve the same data twice. Some of the lobby functions might be possible to cache, but player registration - to take one example - obviously wouldn't work.

 

Or am I missing something?

0

Share this post


Link to post
Share on other sites

Well even if the results are personalized cloudflare has more ddos protection than 1 person with limited funding and limited time can do and caching could still be of use just not as much for the personalized results but things like some of the lobbys and player registration(until posted) could be and wouldnt have to touch your server at all. I run quite a few websites and trying to get into the gaming world on my spare time so my answers will definitly be web biased :) but running my own servers cloudflare has opened many doors that would have cost much more and much more time to handle on my own.
 

And like I mentioned it can handle the ssl so your server doesnt have to which is a major plus because you can quickly kill the server resources with multiple connections on a ddos and it would be much harder to take down a network like cloudflare.

 

What kind of game are you making is it a mmo or something like fps? Some mmos work entirely over web traffic and some fps use ssl web traffic to login and get going like firefall they use aws.

 

I personally feel like if its a fps your better off getting high powered servers and trying managing the downtime that comes from my team fortress and css servers theres just not much you can do other than not making it worth the attack.

0

Share this post


Link to post
Share on other sites

Lobby and Game appear to be on the net to me. If you're forwarding to them, they're on the net.

The typical behavior is to have a proxy server that sits between the servers that do the game logic and the player. The proxy server does all the validation, load balancing, packet validation, etc. Then it forwards the player requests on to the game logic servers. Those servers do the processing of things like physics, making sure that the actions are acceptable, and then issue a response to the proxy server that tells it aye or nay to the player's request, along with publishing relevant state information.


That sounds like a lot of work unless validation/packet validation is heavyweight - and that should depend a lot on the type of game. I personally worked on poker server backends for a few years and I never ran into a need to do something like that.
 
That said, the same company worked on a "next gen" poker platform which distributed everything, even worse than what you describe. Scalability and reliability was in the dust for that platform.
 
So to me that solution sounds like quite the trade-off.
 
I'm also wondering what sort of attack you'd do on a game server with a single open port. Yes, it's possible to overload it, but killing the proxy by overloading it would amount to the same thing, wouldn't it? Please explain the benefits, because I don't really see them.


Packet validation, i.e. verifying CRCs, packet types, etc. and then translating those into game events is fairly lightweight, and the perfect job for a proxy device. It allows you to have a buffer between the players and the game play servers, which allows you to invisibly scale game play servers based on CPU load, and add additional proxies to handle network load.

Because the proxy must be able to send data out, one could use a remote code execution exploit on the open incoming port to launch a process that establishes an outgoing connection to a remote server. At that point I now have the ability to remotely control the proxy server.

Once I've got the ability to remotely control the proxy server, TO ANY DEGREE AT ALL, I can then work towards moving laterally on the network to nearby devices. This includes things like using the database credentials of the server to establish connections to the database and pulling down the data off of it. Since your database server wasn't nicely hidden on another part of the network, and because database access wasn't performed through some sort of service, I've now obtained a list of all the usernames, passwords, email addresses, and all that other information that traditionally gets leaked. But wait, there's more!

Let us imagine for a second that this is a F2P game, money is made by the company by selling certain perks. Since I now have direct DB access I can use that to perform actions such as adding certain items to my account. If this was a game like EVE Online that action could have significant in game economic impact, along with the potential realworld economic impact on THE COMPANY. Imagine adding a thousand PLEX to a single account and selling them on the EVE market. You could trivially depress the market for plex well below the current amount (something like 600,000,000 ISK), and also the company would lose $17,500.

Furthermore, since this would all be done through a few simple SQL queries, you would have a hell of a time tracking it down and even finding out WHICH accounts had been altered, assuming you even noticed it.

The proxy server is your first line of defense, and it should be setup in such a way as to assume that it WILL be compromised, and from there you design the system such that said compromised system cannot significantly affect the rest of your systems. You do this by isolating the network it is on (typically as a DMZ) and only allowing it to talk to specific servers on the internal net, such as game play servers. This then means that in order for me to penetrate further than a proxy server I now have to engineer an exploit that will hit the game play servers. Furthermore, since you can very explicitly restrict exactly what ports are open for Incoming AND OUTGOING connections on game play servers (which you can't do on the proxy servers, as it must be able to maintain connections to a disparate set of clients) you can limit the ability of the hacker to send and receive from those systems (they would have to inject the code into the current process or crash the game play service, both of which would likely be noticed).
1

Share this post


Link to post
Share on other sites


Packet validation, i.e. verifying CRCs, packet types, etc. and then translating those into game events is fairly lightweight, and the perfect job for a proxy device. It allows you to have a buffer between the players and the game play servers, which allows you to invisibly scale game play servers based on CPU load, and add additional proxies to handle network load.

Well, that depends on the game again. Depending on the setup, you might still be able to scale up game play servers, even without a proxy.

 

This was already something with could do on the poker servers I mentioned. Basically tables would spawn on all available game server, auto-balancing themselves so the server with the lowest load took on the most new tables. If you added a new game server it would report itself to the lobby and start creating tables. Removing a server was as simple as telling the server not to create any more tables.

 

This was a very straightforward solution and consequently had very few issues.

 

Similarly in my game, each game server is really game agnostic. Game worlds are persisted to a db, but held entirely in memory once a game server start "hosting" it. Once gameplay winds down the new state can again be persisted to db. Again, this is very domain specific to my particular game as play is done in short sessions and each game only holds about 100 players.

 

But I can easily see games that your proxy solution could work great for too.

 


Because the proxy must be able to send data out, one could use a remote code execution exploit on the open incoming port to launch a process that establishes an outgoing connection to a remote server. At that point I now have the ability to remotely control the proxy server.

 

Let's say the firewall is a NetBSD installation forwarding ports 6000, 6100, 6200 to three different machines on the local network. This machine again is firewalled from the local network except for the game ports. Typically it's only possible to reach the firewall using ssh from a special machine, everything else is blocked.

 

It doesn't seem like a trivial hack, breaking into the servers hosting the games (and from those, reaching the db) 

0

Share this post


Link to post
Share on other sites

Let's say the firewall is a NetBSD installation forwarding ports 6000, 6100, 6200 to three different machines on the local network. This machine again is firewalled from the local network except for the game ports. Typically it's only possible to reach the firewall using ssh from a special machine, everything else is blocked.
 
It doesn't seem like a trivial hack, breaking into the servers hosting the games (and from those, reaching the db)


All I have to do is find a packet combination that includes the code I want to execute along with the appropriate buffer overflow exploit. Its not impossible to do, and happens frequently enough on the Internets. That's how a majority of the hacks over the last few years have worked. Exploiting bugs in software to gain remote access to the machine.

Simply fire walling your game play servers is not sufficient. The servers can still establish outgoing connections on various ports, and if you attempted to prevent that you would find, very quickly, you were unable to have players logging into the machine, especially if you were using TCP/IP.

Lateral movement, once I'm on your network, is actually very easy. It is, in fact, easier than getting into the initial machine in most cases. This is because your internal network has less security on it than the gateway will. Leaves me a significantly greater number of potential access paths. Furthermore, since your game play server has to access the database, I simply need to access the configuration file and I've got your database login details. Now I can simply download the database from the server with those credentials.
0

Share this post


Link to post
Share on other sites

Washu:

So your point is that with a proxy I can lock the game servers to a single outbound port which is a big security win, right?

 

You're basically thinking of an exploit that can go right through the firewall and execute arbitrary code on the machines hosting the game servers. But if you are, then locking the game servers shouldn't be much of an issue. It's a few more jumps to attack the service hosting the db and then onto the db, but there's not really any new kind of obstacle added, is there?

 

But I am assuming a breach so:

 

1. Login does not depend on permanent passwords, when you log in with an existing account you need to verify the login using your e-mail. You're the issued a new temporary passcode for that device. In case of a breach, all passcodes can be reset server side with the only inconvenience that the players need to re-authorize. No passwords are leaked.

 

2. Game server certificates and keys are stored encrypted in the config files, so even with the config files the private keys can't be recovered.

 

3. Even if the keys are leaked, changing them will only invalidate issued tickets.

 

4. If the game server private key is leaked that will only enable a MitM attack, as the keys are only used to authenticate the server to a player.

 

Not perfect, but better than having user passwords in the db.

0

Share this post


Link to post
Share on other sites

It's a few more jumps to attack the service hosting the db and then onto the db, but there's not really any new kind of obstacle added, is there?

Its a lot harder than you think. With the game services exposed via a public port then I have a direct attack vector. I.e. the port. If I have to run everything through the proxy server then it will take longer, and you're more likely to notice.
 

2. Game server certificates and keys are stored encrypted in the config files, so even with the config files the private keys can't be recovered.

Encrypted how? If i'm ON the server then I can read anything the SERVER can read. I can also poke around in memory and simply grab the decrypted keys, or even skip that and grab the decrypted configuration file.
 

4. If the game server private key is leaked that will only enable a MitM attack, as the keys are only used to authenticate the server to a player.

And a MitM attack is exactly what I would want, if i didn't have access to the database. As it would let me collect all the logins going to the server. For a GOLD FARMER, that's what I want. I liquidate all the assets on your account, and then sell the gold to some sucker with too much money. EVE had that issue for a while, so does WoW. Mind you, their servers weren't hacked (and their design matches what I described btw), but instead they stole the credentials using viruses or spoofed websites. Edited by Washu
1

Share this post


Link to post
Share on other sites

Let me try and explain this a bit better:

If you have a set of game servers that are all exposed to the internet (port forward, just behind a firewall, on the DMZ, whatever). These servers, obviously, have to talk to the database. So clearly there's an open connection between them and the database. Even if the database is behind another firewall, on a different network segment, or any number of other network design decisions.

But let us take the example of a set of port forwards to the different game servers, with the database server on the same network as the game servers.

If all of the game servers are on the same network then, once one is compromised, then it becomes fairly trivial to move laterally to all the other machines *on that segment of the network*. I.e. if I compromise the server at port 6000, I have control over that machine. I can now establish LOCAL connections from that machine to all of the other machines on that network segment, INCLUDING the database server.

The chances of you detecting and stopping such an intrusion in time to prevent damage, or the theft of data, is rather minuscule.

Once someone is ON your servers, no amount of encryption will help you. They can poke directly into memory to yank the keys, passwords, algorithms, or anything else they want out. They can decrypt your configuration files and database connection strings with minimal effort. Frankly, once they're on that segment of the network, that segment is compromised and NOTHING on it is guaranteed.

Now, if we change the network layout slightly, we end up improving security immensely. Specifically, if we move the database server off the local network and onto an internal LAN, then setup a firewall between that LAN and the DMZ that the game servers occupy we have introduced an additional barrier. No longer can I simply remote straight into the database server and say, make a copy of the raw database file for later examination. I'm reliant on connecting via the game servers into the database and running queries. Those queries will only have the permissions of the user that is used by the game servers, although those permissions are usually pretty wide since the game servers have to write changes to the database. Nevertheless, this introduces an additional step, and increases the chances of my detection significantly. Any serious queries I write will introduce a noticeable database load, and it makes it a lot harder for me to simply disguise my hacking as random crap that's getting thrown at the server. While this is good, its not great. The chances of detection are still really low, and I can still end up doing bulk queries and getting large amounts of data out of the system before you ever notice.

Now, if we add an intermediary between the client and the game servers we can impose additional layers of security. In addition we get additional scalability options. Firstly, if we have a DMZ with a series of proxy servers on it, with those proxy servers doing nothing but translating client packets into game events, which are send to the game servers, then we can scale the networking side of things (provided we have a decent backbone between the game servers and the proxies), fairly easily. Too much traffic for one proxy to handle? Toss up another and tell the login server about it.

The game servers are then put behind the firewall with the database. The only way in or out of the local area network is via communication with the game servers. The database is not exposed to the proxy servers at all. Since the game servers only communicate with the proxy servers via game events (i.e. internal packets), then the only ways to attack the game servers, and thus gain direct access to the database, would be to work out YET ANOTHER EXPLOIT and inject that into one of the game servers. In addition, since the game servers only respond to game events, I cannot issue bulk queries and obtain copies of your database WITHOUT hacking my way through that barrier.

Now, if your proxies are doing the validation and translation of packets, batching up zone events and sending them to the correct game server, etc. Then your game servers should be doing validation of those events, such as running physics, etc. This does not exempt the game servers from running basic validation, such as ensuring packet format and length are correct for the event type specified.

This significantly increases the chances of detection. Something as simple as trying a buffer overflow on say a player position update request would likely be caught. Too many such requests might ban that account, but if you kept seeing those requests in the admin log, you would probably open up your sniffer and see whose running a bot.

Now, its not perfect. Someone can STILL get in, and you can still not notice them. But the barrier to entry has been significantly increases with relatively little cost. For any REAL MMO you're going to have to have multiple servers running a single shard ANYWAYS.

As a reference, here is the architecture of the EVE node system, the VPN links are firewalled (i.e. DMZ):

e2lm.png

(This is from their GDC 2009 presentation)

Edited by Washu
2

Share this post


Link to post
Share on other sites


Its a lot harder than you think. With the game services exposed via a public port then I have a direct attack vector. I.e. the port. If I have to run everything through the proxy server then it will take longer, and you're more likely to notice.

Definitely, and we could keep adding different proxies between the entry point and the game to make it even harder to find an exploit.

 

I'm not saying that this is bad, just that the cost in development/architecture/latency has to be balanced against the gain for hacker to access the system.

 


Encrypted how? If i'm ON the server then I can read anything the SERVER can read. I can also poke around in memory and simply grab the decrypted keys, or even skip that and grab the decrypted configuration file.

 

Oh, the config is in plain text, but the particular value of the key needs to be decrypted to work. It's decrypted in memory of course, but since the key is simply a random value, you'd probably want to grab the java classes and reverse engineer those first in order to see where that's stored. Again, not impossible, but adds at least a day of work (for someone who's already reverse-engineered the entire protocol).

 

Since generating a new cert is trivial - it's then simply a matter of pushing out new clients - an attacker would need to work fast to put it to any use. Once a breach is discovered, changing cert + resetting passcodes removes any future gain for the attacker. Since the passcodes are just random numbers, the only permanent gain an attacker would have would be if they changed account data.

 

Again, for a different game this would be a much greater problem.

0

Share this post


Link to post
Share on other sites

Washu

 

Thanks, that was a very interesting rundown of possible sploits (lots of material for a FAQ or good article there).

 

I'm still going to insist on saying that you are assuming a certain problem domain here. In particular a MMO in the style of EVE or WoW.

 

In my case:

 

- Players do not select (nor have any easy way to identify) the game instance/world

- Players' names are anonymized

- Players cannot exchange in-game currency with each other

- Each instance only holds about 100 player

- Accounts without user-selected passwords

- No real money transactions within the system

 

The only gain in breaching security would be cheating, and a single cheater is unlikely to cheat in more than a few worlds, affecting at the most a few hundred players.

 

What I do want to protect against is cheating that can be discovered and utilized by a large amount of players with little effort.

 

One of the strategies I have to combat that is that instead of - as is traditional - validate the command before performing it, I simply have checks built into every single game value which is validated every time they change. Failing a validation throws an error that rolls back any changes done during the execution of the command.

 

That way, instead of having to remember to check everything up front, I simply perform all actions and rely on that the objects themselves will throw errors when the player attempts something illegal. This ensures atomicity regardless of error type.

0

Share this post


Link to post
Share on other sites

One thing of note however. Be sure that your product's architecture does not confuse certain legal user actions as DDoS attacks. We've had that where I work once where, in trying to mitigate severe DDoS against one of our products, we've also inadvertently gone too far and ended up disrupting service for some 'core users' whose actions per minute were well above established standards...

0

Share this post


Link to post
Share on other sites


Well even if the results are personalized cloudflare has more ddos protection than 1 person with limited funding and limited time can do and caching could still be of use just not as much for the personalized results but things like some of the lobbys and player registration(until posted) could be and wouldnt have to touch your server at all. I run quite a few websites and trying to get into the gaming world on my spare time so my answers will definitly be web biased but running my own servers cloudflare has opened many doors that would have cost much more and much more time to handle on my own.

 

Cloudflare is awesome for web, but of limited use when you deal with plain sockets.

 


And like I mentioned it can handle the ssl so your server doesnt have to which is a major plus because you can quickly kill the server resources with multiple connections on a ddos and it would be much harder to take down a network like cloudflare.

 

Yeah, that's actually the reason for me having a dedicated login server. I brought my dev servers to their knees by selecting a naive RSA key size and reconnecting a few hundred bots in the span of a few seconds.

 

Now I have two stops in place:

 

1. Before the login server even looks begins doing any CPU intensive work, it gives you a puzzle challenge ("find a value, which, when hashed with this random value, produces a result with at least n trailing zeroes"). The better you answer, the higher priority in the login queue (and a incorrect result puts you last). A client which places outside of the queue has to request a new puzzle.

 

2. From the login server you get a ticket, which the game/login servers can open up with almost zero cost. This lets the server setup a secure channel without an asymmetric key exchange.

 

This allows me to minimize and constrain the CPU usage. Obviously, just using Cloudflare with SSL would be a whole lot simpler.

 


What kind of game are you making is it a mmo or something like fps? Some mmos work entirely over web traffic and some fps use ssl web traffic to login and get going like firefall they use aws.

 

It's an unconventional mmo, more like a multiplayer strategy mixed with rpg. Imagine, perhaps, a multiplayer "Pirates!" in a fantasy world. The most accurate description would be "Trade Wars" (the BBS door) played on a fantasy map/world.

 

In terms of responsiveness, I'm now seriously considering using UDP because I need low latency map movement (or the game will feel sluggish) and I have no real way to do anything predictive on the client side. A roundtrip in the vicinity of 100-200 ms on 3G is what I'm aiming for. This is nothing I would reliably get over HTTP or even TCP/IP due to the frequency of mobile wifi/3G packet drops.

0

Share this post


Link to post
Share on other sites


If you have a set of game servers that are all exposed to the internet (port forward, just behind a firewall, on the DMZ, whatever). These servers, obviously, have to talk to the database. So clearly there's an open connection between them and the database. Even if the database is behind another firewall, on a different network segment, or any number of other network design decisions.

 

I just want to mention that I do think this is an interesting solution, but to be worth implementing for my particular game I would need to see some benefit beyond increased resistance against security breaches.

 

In my setup with login/lobby/game servers being hosted on different servers, I know that hosing the login server will not affect players already with a login ticket. Since such a ticket is valid 24h (depending on setting), a DDoS against the login servers would be easy to handle.

 

However, with the proxies, targets drop from 3 different to one single type (the proxy). In order to DDoS, it's now only necessary to hose the proxy.

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0