Jump to content

  • Log In with Google      Sign In   
  • Create Account


Game server DoS / DDoS mitigation strategies?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
29 replies to this topic

#1 lerno   Members   -  Reputation: 207

Like
0Likes
Like

Posted 02 August 2013 - 01:51 PM

Are there any must-have DoS / DDoS mitigation strategies one should always build into a server?

 

Regardless of TCP or UDP, it feels like there is very little one could do against a DDoS which tries to saturate the network.

 

Even if the attacker can't do that, simply sending login packets with spoofed IP addresses/ports could be a problem. For something like SSL, it's possible to hit the CPU hard by initiating a handshake. CPU exhaustion attacks can be mitigated by puzzle challenge-style logins, but it should be fairly easy to block login by making sure that the all login "slots" are in use.

 

(I know some websites that use CloudFlare, but that's for serving http/https content and so isn't an option)

 

Any opinions of what a reasonable line of defence is?



Sponsor:

#2 AllEightUp   Moderators   -  Reputation: 4122

Like
4Likes
Like

Posted 02 August 2013 - 06:44 PM

There are always some things you should do in server code but they are only going to handle relatively small DoS attempts.  Higher bandwidth multi-location DDoS attempts are something you have to plan for using colo's, multiple IP's, hardware, etc.  Assuming just making the server able to withstand general and minor attacks, you should handle several things.  There is a bit of a difference between TCP and UDP though:

 

1.  With TCP you need to check your OS syn flood measures.  See http://en.wikipedia.org/wiki/SYN_flood.  I believe most OS's have many of the protections already in place, you might want to tweak them if possible.  I believe many Linux servers include automatic exponential syn reply backoffs to handle DDoS so you may be about as covered as you can get.  Just a good thing to check.

2.  UDP servers will have much the same sort of flood attack vulnerability, so you have to implement the various bits of mitigation code yourself.

3.  Both TCP and UDP need to have bandwidth limiters.  The attacker could have a valid login but a hacked client or tool which once logged in starts blasting bandwidth at the server beyond what it should.  With TCP you just disconnect them, UDP you do the same thing effectively.

4.  UDP needs to validate each packet is not a spoof/injection.  Generally the first thing in the packet should be some very quick to validate hash/crc value.  The last time I did this I used SSL to get the initial connection from client to server, perform login securely and give the client a unique ID/salt value.  Then, when a UDP game packet is received, the first 16 bits are a crc of the salt, the ip and packet length.  With that, you can at least check the basic validity of a packet cheaply before you start full decode, validation and processing.  Obviously if the attack is coming from someone with a real client or tool which logs in properly they can have the salt value and you might get valid packets that should still be thrown away.  You should boot the bastard if you see such thrown out packets too often of course.

5.  Exponential backoff on failed login attempts.  I.e. bad username/password.  This should be a ip+user recorded item.  So, the first time a given IP+username gives you a bad login, you refuse to accept login for 5 seconds.  Next time, 10 seconds, then 20, etc.  You record it as ip+username incase the real user attempts to connect after the attack.  You don't want the user to have to wait possibly several minutes for something they didn't do.. :)

6.  Same ip+multiple usernames login failures.  Same as #5 basically.

 

These are just some of the larger things I can think of off the top of my head.  There are a large number of additions for UDP connections.  This is not a small subject, it is actually quite large.  But, the basics above are a simple starting selection of the more important ones.



#3 jeff8j   Members   -  Reputation: 679

Like
0Likes
Like

Posted 02 August 2013 - 06:53 PM

With the mentioning of cloudflare and in AllEightUps post number 4 he uses a ssl connection for initial id/salt I saw recently that the new beta firefall uses simple https requests to aws servers for login. You could do similar use cloudflare to prevent alot of attacks on the initial login atleast. You would still have to worry about the bandwidth issues after logged in though. Depending on what kind of game it is if its not real time you could possibly get away with all http requests using the cloudflare or other technology thats already in place.


Firefox youtube video and audio downloader MP3 MP4 OGG WEBM

https://addons.mozilla.org/en-US/firefox/addon/simple-youtube-converter/


#4 hplus0603   Moderators   -  Reputation: 4977

Like
3Likes
Like

Posted 02 August 2013 - 10:36 PM

The ability to detect remote IPs that issue an unnormally high rate of requests, or an unnormal distribution of request types, is useful.

 

Also, hardening your system and setting in place upstream filtering is useful. If your ISP can throw away UDP traffic destined to you on ports that you don't care about, then your link doesn't need to worry about this. (The ability to do this varies based on the ISP and your relation to them, i e how good a customer you are.)

 

Making sure you have SYN cookies turned on on your servers and/or load balancers is common sense. Same thing for filtering obviously bad IP fragments, etc.

 

Regarding CloudFlare, they are a mid-tier DDoS vendor, worrying more about CDN than DDoS, so they don't have a lot of ability to work with custom protocols like games. I think you'd get the same from other CDNs like Akamai.

There are higher-end vendors, where you pay five figures a month plus possibly five figures per event to help mitigate attacks. Those vendors may be able to put in place custom filters that you and they develop together, and can target any services. Names that come to mind include Prolexic, Neustar, and Verisign. Some of those guys even claim that they started out specifically as DDoS mitigation providers for MMO games!

 

Another option is to run on a public cloud, such as Amazon ECC. Yes, you pay for bandwidth, but if the DDoS is some number of gigabits for a few hours, that's not actually going to accrue all that much traffic. On the other hand, on ECC, a determined attacker will be able to generate pretty big bandwidth bills for your service... and Amazon may find that your service disrupts other customers and cut you off if it gets bad enough.

 

Finally, you can get high-speed connections to your ISP, with a lower commit level. For example, you might be able to find an ISP that lets you commit to half a gigabit per second, yet allows 100 Gbps interconnect, and bills by the 95th percentile. You need to be under DDoS for almost two days, full time, to actually make the DDoS hit your 95th percentile. Given that renting out botnets is a lucrative market these days, somebody has to be really pissed at you to hit you with dozens or even a hundred gigabits for more than an hour. I've only really heard of that happening to the financial services people, presumably by black op teams from their competitors or extortionists.


enum Bool { True, False, FileNotFound };

#5 lerno   Members   -  Reputation: 207

Like
0Likes
Like

Posted 03 August 2013 - 01:16 AM

Good ideas all around. Some questions:

 

AllEightUp:

For bandwidth-limiting, you're thinking of a simple thing to prevent the client from issuing requests and getting response from the server for the request, right? Because nothing's going to prevent the packets actually hitting my server. Or are you thinking of some configuration I could do to firewalls?

 

For UDP I have encryption, but only after the UDP payload is spliced into sub-commands. I don't have it on the UDP protocol level. I don't know if that's a problem or not. For TCP the encryption happens after I split the stream into packets, but that feels "safer".

 

jeff8j:

Doing login over https has occurred to me exactly for that reason. Right now it uses a puzzle-challenge and limited evaluation pipe to prevent overload. Still, that won't protect it against bandwidth saturation. On the other hand, say I go with something like aws, then this might even make it worse. After all, killing the login server with requests is fairly harmless. However, if they fail to do so, or are able to get a large amount of login tickets, then they might start targeting the lobby and game servers - and those are the ones I would like to protect.

 

I've made secure login for game/lobby as cheap as possible by relying on a kerberus-style ticket to set up encryption, but they are hitting the db after the security handshake... The nice thing about the login servers are that they don't affect any sub systems if overloaded.

 

It's hard for me to tell if moving the login to a simple https service would help or not.

 

hplus0603:

What would I gain by filtering but my game port? Assuming I have a firewall closing everything else, what would they get by bombarding other ports, as opposed to simply the game port?



#6 Washu   Senior Moderators   -  Reputation: 4486

Like
0Likes
Like

Posted 03 August 2013 - 03:15 AM

There are also other things you need to be sure to handle, one of the latest and greatest attacks is the simplest of all:

 

A pseudo slow client.

 

The trick is to establish TCP connections, valid ones, but send as little data as possible. Since data is trickling over the connection, the bad connection remains open, consuming a great deal of resources. It is, in effect, the same as a TCP Syn flood attack, except the connections actually complete.

 

There are, of course, the other things you need to handle:

 

Validate everything, never trust data coming from the client.

 


What would I gain by filtering but my game port? Assuming I have a firewall closing everything else, what would they get by bombarding other ports, as opposed to simply the game port?

A software firewall on a port can "help" to a degree, but the machine still ends up having to handle the data getting sent to it. While the data is simply discarded, it is still being sent to the machine.

 

Hardware firewalls are a bit better on this, since they're actually designed for throughput and packet streaming, so they can tend to discard blocked traffic at a much faster rate, but again... that's still traffic ON THE WIRE. If you're getting charged for bandwidth, you'll get charged regardless of if you accepted the traffic or not.


I've made secure login for game/lobby as cheap as possible by relying on a kerberus-style ticket to set up encryption, but they are hitting the db after the security handshake... The nice thing about the login servers are that they don't affect any sub systems if overloaded.

 

Who is hitting the database? The client should never touch the database. In fact, the service the client talks to should, ideally, not talk directly to the database. The database should be firewalled, locked away behind services that do all the intermediate actions. Those services should be located on servers that are locked away from the rest of your machines, if at all possible. The simple fact is, a server on the internet is a gateway. Once someone gets into your gateway, then you need to limit their lateral movement. A database server sitting on the same network as said gateway is a ripe target to get its data ripped. But a database server locked behind some service gateways and  firewalls is a much harder target to access and would give you time to notice and stop said infiltration.


In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.
ScapeCode - Blog | SlimDX


#7 lerno   Members   -  Reputation: 207

Like
0Likes
Like

Posted 03 August 2013 - 03:48 AM

Washu: In regards to the port blocking, say I have the game port at 6000. What does an attacker gain by flooding 6001 that the attacker can't gain by flooding 6000? That's what I was wondering.

 

In regards to the db - it's the server that connects to the db of course, I was referring to the fact that a login / request by the client may cause the server to perform a db query.

 

In the case of a DoS attack on a server, then that server will trigger many db requests. If the queries are good and the server has a connection pool that limits the number of simultaneous queries to the db, that shouldn't be a problem. Still, I prefer to avoid enabling a player to cause a db query as far as possible.

 

Like you said, the db should never be accessible directly from the internet, and preferably thoroughly locked away inside the internal network.

 

In my case I was imagining the setup looking a bit like this:

 

 

 
              Internet
                  |
                  |
                  V 
            +----------+
            | Firewall |  some.domainnamehere.net <- Firewall connected to internet
            +----------+     
      6000    |  |  |    6200               Port forwarding to the right server
   ___________|  |  |____________
   |             |              |
   |             | 6100         |
   |             |              |
   v             v              v
+-------+    +-------+      +------+
| Login |    | Lobby |      | Game |   <- Servers not directly connected to net
+-------+    +-------+      +------+
                 |              |
                 |_____    _____|
                      |    |
                      v    v
                    +--------+
                    |   Db   |
                    +--------+

Edited by lerno, 03 August 2013 - 03:49 AM.


#8 Washu   Senior Moderators   -  Reputation: 4486

Like
1Likes
Like

Posted 03 August 2013 - 08:35 AM

Washu: In regards to the port blocking, say I have the game port at 6000. What does an attacker gain by flooding 6001 that the attacker can't gain by flooding 6000? That's what I was wondering.

Not much, other than increasing your traffic bill. They're much more likely to attempt to flood an open port than one that's closed. Mainly because the goal is to swamp the machine, and the easiest way to swamp a machine is to use up the kernel's free memory by clogging it up with useless garbage through open ports.
 

In regards to the db - it's the server that connects to the db of course, I was referring to the fact that a login / request by the client may cause the server to perform a db query.
 
In the case of a DoS attack on a server, then that server will trigger many db requests. If the queries are good and the server has a connection pool that limits the number of simultaneous queries to the db, that shouldn't be a problem. Still, I prefer to avoid enabling a player to cause a db query as far as possible.
 
Like you said, the db should never be accessible directly from the internet, and preferably thoroughly locked away inside the internal network.
 
In my case I was imagining the setup looking a bit like this:

Lobby and Game appear to be on the net to me. If you're forwarding to them, they're on the net.

The typical behavior is to have a proxy server that sits between the servers that do the game logic and the player. The proxy server does all the validation, load balancing, packet validation, etc. Then it forwards the player requests on to the game logic servers. Those servers do the processing of things like physics, making sure that the actions are acceptable, and then issue a response to the proxy server that tells it aye or nay to the player's request, along with publishing relevant state information.

There would be a DMZ for the proxy servers, and then an internal network for the game logic servers.

In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.
ScapeCode - Blog | SlimDX


#9 AllEightUp   Moderators   -  Reputation: 4122

Like
1Likes
Like

Posted 03 August 2013 - 09:47 AM

AllEightUp:

For bandwidth-limiting, you're thinking of a simple thing to prevent the client from issuing requests and getting response from the server for the request, right? Because nothing's going to prevent the packets actually hitting my server. Or are you thinking of some configuration I could do to firewalls?

 

For UDP I have encryption, but only after the UDP payload is spliced into sub-commands. I don't have it on the UDP protocol level. I don't know if that's a problem or not. For TCP the encryption happens after I split the stream into packets, but that feels "safer".

 

The bandwidth limiting is basically a preventative measure.  A common attack vector is for a client to login correctly and everything looks nice but they start sending valid commands as fast as possible.  For instance, say you have movement messages, instead of sending a movement message packed in with other data at say 10 times a second, they start sending just the movement update alone at 1000 times a second.  Everything is 'valid' but they are hitting you in two places, the bandwidth of course and the CPU overhead of dealing with the valid but excessive packets.  When detected, you boot the connection and stop accepting the packets, that brings CPU/memory on the server back to normal even if you are still getting slammed with the excessive bandwidth.  If they keep it up even if you keep booting and ignoring them, normal mitigation such as telling the router to bounce the packets for an hour and of course contacting the source ISP or backbone provider and informing them of the ip source etc.

 

As to the encryption, I don't generally trust myself to get all the edge cases correct and rather use the separate SSL connection for initialization.  The ssl connection is only used for login and other secure items.  As a benefit, you can use it for the initial NAT traversal since you have the connection anyway.  In general though, this tcp connection is only there for a brief period and won't interfere with the UDP once initiated.  This is mostly paranoia on my part, I just don't want to risk that I goof something up which allows someone to crack the UDP encryption somehow and grab up everyone's passwords.  It's not paranoid if everyone really is out to get ya... :)



#10 jeff8j   Members   -  Reputation: 679

Like
0Likes
Like

Posted 03 August 2013 - 11:52 AM

Lerno:

Using https wouldnt protect the game servers directly but could do woners as far as login and lobby systems. With the pro cloudflare you can set the cache to 30 seconds and they have servers all over the world so when people hi the lobby they can bring up the cached version 30 seconds isnt that much of a difference then the cloudflare servers would only touch your server once every 30 seconds. You would probably want to whitelist the cloudflare servers and block anything else through a firewall. Also the nameservers would point to cloudflare not your ip so you can keep your ip for login/lobby not known to the public. You could also have a fall back server or multiple if some get in trouble. You would also have the servers report over https to keep that login/lobby server masked. Overhead would also drop as you dont have to serve https from your server you could serve http and have cloudflare do all the heavy https work.

 

Note: If your game could use websockets you could block ips and cloudflares ddos would still work to prevent them from even touching your servers. This does have the downside of its not a direct connection so latency will go up a bit but I have done this and its not that bad I figure because cloudflare trys to be as close to the end user as possible so its pretty much in line on the way to the server.

 

In the end the only way to be slightly confident that things wont go down is to have enough servers in enough locations with big pipes that it would be very difficult to bring them down and if one goes down another handles the load. Thats how google, facebook and the like handle things and thats why a web option is easier to get going something like cloudflare has many powerful servers in place around the world with massive pipes I know I couldnt afford.

 

Of course there are alternatives to cloudflare some way better but none at the same level of cost $20 a month or even free for basic protection so thats why I keep going back to it.


Firefox youtube video and audio downloader MP3 MP4 OGG WEBM

https://addons.mozilla.org/en-US/firefox/addon/simple-youtube-converter/


#11 lerno   Members   -  Reputation: 207

Like
0Likes
Like

Posted 03 August 2013 - 01:55 PM


Lobby and Game appear to be on the net to me. If you're forwarding to them, they're on the net.

The typical behavior is to have a proxy server that sits between the servers that do the game logic and the player. The proxy server does all the validation, load balancing, packet validation, etc. Then it forwards the player requests on to the game logic servers. Those servers do the processing of things like physics, making sure that the actions are acceptable, and then issue a response to the proxy server that tells it aye or nay to the player's request, along with publishing relevant state information.

That sounds like a lot of work unless validation/packet validation is heavyweight - and that should depend a lot on the type of game. I personally worked on poker server backends for a few years and I never ran into a need to do something like that.

 

That said, the same company worked on a "next gen" poker platform which distributed everything, even worse than what you describe. Scalability and reliability was in the dust for that platform.

 

So to me that solution sounds like quite the trade-off.

 

I'm also wondering what sort of attack you'd do on a game server with a single open port. Yes, it's possible to overload it, but killing the proxy by overloading it would amount to the same thing, wouldn't it? Please explain the benefits, because I don't really see them.



#12 lerno   Members   -  Reputation: 207

Like
0Likes
Like

Posted 03 August 2013 - 02:06 PM


As to the encryption, I don't generally trust myself to get all the edge cases correct and rather use the separate SSL connection for initialization.  The ssl connection is only used for login and other secure items.  As a benefit, you can use it for the initial NAT traversal since you have the connection anyway.  In general though, this tcp connection is only there for a brief period and won't interfere with the UDP once initiated.

 

That's good advice in general. I've dug deep into cryptography to design a protocol which I feel fairly confident in. Mostly because it's basically an implementation combining two well known protocols. Still, I know it's a risk.

 

(Even if it's broken, I don't actually use any user generated passwords. In addition, gameplay is mostly anonymized. It's not really possible to determine the e-mail of another player, and since game is partitioned into separate worlds, if you get hold of someone's credentials it's unlikely to be someone playing in the same world as you.)



#13 lerno   Members   -  Reputation: 207

Like
0Likes
Like

Posted 03 August 2013 - 02:12 PM


Using https wouldnt protect the game servers directly but could do woners as far as login and lobby systems. With the pro cloudflare you can set the cache to 30 seconds and they have servers all over the world so when people hi the lobby they can bring up the cached version 30 seconds isnt that much of a difference then the cloudflare servers would only touch your server once every 30 seconds. You would probably want to whitelist the cloudflare servers and block anything else through a firewall. Also the nameservers would point to cloudflare not your ip so you can keep your ip for login/lobby not known to the public. You could also have a fall back server or multiple if some get in trouble. You would also have the servers report over https to keep that login/lobby server masked. Overhead would also drop as you dont have to serve https from your server you could serve http and have cloudflare do all the heavy https work.

 

In my case all of login/lobby/game are plain sockets. No http. That said, it would be possible to convert them to http(s) services. However, cached data would not do. Both login and lobby servers serve personalized results, and for security reasons the login server shouldn't serve the same data twice. Some of the lobby functions might be possible to cache, but player registration - to take one example - obviously wouldn't work.

 

Or am I missing something?



#14 jeff8j   Members   -  Reputation: 679

Like
0Likes
Like

Posted 03 August 2013 - 02:24 PM

Well even if the results are personalized cloudflare has more ddos protection than 1 person with limited funding and limited time can do and caching could still be of use just not as much for the personalized results but things like some of the lobbys and player registration(until posted) could be and wouldnt have to touch your server at all. I run quite a few websites and trying to get into the gaming world on my spare time so my answers will definitly be web biased :) but running my own servers cloudflare has opened many doors that would have cost much more and much more time to handle on my own.
 

And like I mentioned it can handle the ssl so your server doesnt have to which is a major plus because you can quickly kill the server resources with multiple connections on a ddos and it would be much harder to take down a network like cloudflare.

 

What kind of game are you making is it a mmo or something like fps? Some mmos work entirely over web traffic and some fps use ssl web traffic to login and get going like firefall they use aws.

 

I personally feel like if its a fps your better off getting high powered servers and trying managing the downtime that comes from my team fortress and css servers theres just not much you can do other than not making it worth the attack.


Firefox youtube video and audio downloader MP3 MP4 OGG WEBM

https://addons.mozilla.org/en-US/firefox/addon/simple-youtube-converter/


#15 Washu   Senior Moderators   -  Reputation: 4486

Like
1Likes
Like

Posted 03 August 2013 - 02:25 PM

Lobby and Game appear to be on the net to me. If you're forwarding to them, they're on the net.

The typical behavior is to have a proxy server that sits between the servers that do the game logic and the player. The proxy server does all the validation, load balancing, packet validation, etc. Then it forwards the player requests on to the game logic servers. Those servers do the processing of things like physics, making sure that the actions are acceptable, and then issue a response to the proxy server that tells it aye or nay to the player's request, along with publishing relevant state information.


That sounds like a lot of work unless validation/packet validation is heavyweight - and that should depend a lot on the type of game. I personally worked on poker server backends for a few years and I never ran into a need to do something like that.
 
That said, the same company worked on a "next gen" poker platform which distributed everything, even worse than what you describe. Scalability and reliability was in the dust for that platform.
 
So to me that solution sounds like quite the trade-off.
 
I'm also wondering what sort of attack you'd do on a game server with a single open port. Yes, it's possible to overload it, but killing the proxy by overloading it would amount to the same thing, wouldn't it? Please explain the benefits, because I don't really see them.


Packet validation, i.e. verifying CRCs, packet types, etc. and then translating those into game events is fairly lightweight, and the perfect job for a proxy device. It allows you to have a buffer between the players and the game play servers, which allows you to invisibly scale game play servers based on CPU load, and add additional proxies to handle network load.

Because the proxy must be able to send data out, one could use a remote code execution exploit on the open incoming port to launch a process that establishes an outgoing connection to a remote server. At that point I now have the ability to remotely control the proxy server.

Once I've got the ability to remotely control the proxy server, TO ANY DEGREE AT ALL, I can then work towards moving laterally on the network to nearby devices. This includes things like using the database credentials of the server to establish connections to the database and pulling down the data off of it. Since your database server wasn't nicely hidden on another part of the network, and because database access wasn't performed through some sort of service, I've now obtained a list of all the usernames, passwords, email addresses, and all that other information that traditionally gets leaked. But wait, there's more!

Let us imagine for a second that this is a F2P game, money is made by the company by selling certain perks. Since I now have direct DB access I can use that to perform actions such as adding certain items to my account. If this was a game like EVE Online that action could have significant in game economic impact, along with the potential realworld economic impact on THE COMPANY. Imagine adding a thousand PLEX to a single account and selling them on the EVE market. You could trivially depress the market for plex well below the current amount (something like 600,000,000 ISK), and also the company would lose $17,500.

Furthermore, since this would all be done through a few simple SQL queries, you would have a hell of a time tracking it down and even finding out WHICH accounts had been altered, assuming you even noticed it.

The proxy server is your first line of defense, and it should be setup in such a way as to assume that it WILL be compromised, and from there you design the system such that said compromised system cannot significantly affect the rest of your systems. You do this by isolating the network it is on (typically as a DMZ) and only allowing it to talk to specific servers on the internal net, such as game play servers. This then means that in order for me to penetrate further than a proxy server I now have to engineer an exploit that will hit the game play servers. Furthermore, since you can very explicitly restrict exactly what ports are open for Incoming AND OUTGOING connections on game play servers (which you can't do on the proxy servers, as it must be able to maintain connections to a disparate set of clients) you can limit the ability of the hacker to send and receive from those systems (they would have to inject the code into the current process or crash the game play service, both of which would likely be noticed).

In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.
ScapeCode - Blog | SlimDX


#16 lerno   Members   -  Reputation: 207

Like
0Likes
Like

Posted 03 August 2013 - 02:55 PM


Packet validation, i.e. verifying CRCs, packet types, etc. and then translating those into game events is fairly lightweight, and the perfect job for a proxy device. It allows you to have a buffer between the players and the game play servers, which allows you to invisibly scale game play servers based on CPU load, and add additional proxies to handle network load.

Well, that depends on the game again. Depending on the setup, you might still be able to scale up game play servers, even without a proxy.

 

This was already something with could do on the poker servers I mentioned. Basically tables would spawn on all available game server, auto-balancing themselves so the server with the lowest load took on the most new tables. If you added a new game server it would report itself to the lobby and start creating tables. Removing a server was as simple as telling the server not to create any more tables.

 

This was a very straightforward solution and consequently had very few issues.

 

Similarly in my game, each game server is really game agnostic. Game worlds are persisted to a db, but held entirely in memory once a game server start "hosting" it. Once gameplay winds down the new state can again be persisted to db. Again, this is very domain specific to my particular game as play is done in short sessions and each game only holds about 100 players.

 

But I can easily see games that your proxy solution could work great for too.

 


Because the proxy must be able to send data out, one could use a remote code execution exploit on the open incoming port to launch a process that establishes an outgoing connection to a remote server. At that point I now have the ability to remotely control the proxy server.

 

Let's say the firewall is a NetBSD installation forwarding ports 6000, 6100, 6200 to three different machines on the local network. This machine again is firewalled from the local network except for the game ports. Typically it's only possible to reach the firewall using ssh from a special machine, everything else is blocked.

 

It doesn't seem like a trivial hack, breaking into the servers hosting the games (and from those, reaching the db) 



#17 Washu   Senior Moderators   -  Reputation: 4486

Like
0Likes
Like

Posted 03 August 2013 - 03:18 PM

Let's say the firewall is a NetBSD installation forwarding ports 6000, 6100, 6200 to three different machines on the local network. This machine again is firewalled from the local network except for the game ports. Typically it's only possible to reach the firewall using ssh from a special machine, everything else is blocked.
 
It doesn't seem like a trivial hack, breaking into the servers hosting the games (and from those, reaching the db)


All I have to do is find a packet combination that includes the code I want to execute along with the appropriate buffer overflow exploit. Its not impossible to do, and happens frequently enough on the Internets. That's how a majority of the hacks over the last few years have worked. Exploiting bugs in software to gain remote access to the machine.

Simply fire walling your game play servers is not sufficient. The servers can still establish outgoing connections on various ports, and if you attempted to prevent that you would find, very quickly, you were unable to have players logging into the machine, especially if you were using TCP/IP.

Lateral movement, once I'm on your network, is actually very easy. It is, in fact, easier than getting into the initial machine in most cases. This is because your internal network has less security on it than the gateway will. Leaves me a significantly greater number of potential access paths. Furthermore, since your game play server has to access the database, I simply need to access the configuration file and I've got your database login details. Now I can simply download the database from the server with those credentials.

In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.
ScapeCode - Blog | SlimDX


#18 lerno   Members   -  Reputation: 207

Like
0Likes
Like

Posted 03 August 2013 - 03:43 PM

Washu:

So your point is that with a proxy I can lock the game servers to a single outbound port which is a big security win, right?

 

You're basically thinking of an exploit that can go right through the firewall and execute arbitrary code on the machines hosting the game servers. But if you are, then locking the game servers shouldn't be much of an issue. It's a few more jumps to attack the service hosting the db and then onto the db, but there's not really any new kind of obstacle added, is there?

 

But I am assuming a breach so:

 

1. Login does not depend on permanent passwords, when you log in with an existing account you need to verify the login using your e-mail. You're the issued a new temporary passcode for that device. In case of a breach, all passcodes can be reset server side with the only inconvenience that the players need to re-authorize. No passwords are leaked.

 

2. Game server certificates and keys are stored encrypted in the config files, so even with the config files the private keys can't be recovered.

 

3. Even if the keys are leaked, changing them will only invalidate issued tickets.

 

4. If the game server private key is leaked that will only enable a MitM attack, as the keys are only used to authenticate the server to a player.

 

Not perfect, but better than having user passwords in the db.



#19 Washu   Senior Moderators   -  Reputation: 4486

Like
1Likes
Like

Posted 03 August 2013 - 04:10 PM

It's a few more jumps to attack the service hosting the db and then onto the db, but there's not really any new kind of obstacle added, is there?

Its a lot harder than you think. With the game services exposed via a public port then I have a direct attack vector. I.e. the port. If I have to run everything through the proxy server then it will take longer, and you're more likely to notice.
 

2. Game server certificates and keys are stored encrypted in the config files, so even with the config files the private keys can't be recovered.

Encrypted how? If i'm ON the server then I can read anything the SERVER can read. I can also poke around in memory and simply grab the decrypted keys, or even skip that and grab the decrypted configuration file.
 

4. If the game server private key is leaked that will only enable a MitM attack, as the keys are only used to authenticate the server to a player.

And a MitM attack is exactly what I would want, if i didn't have access to the database. As it would let me collect all the logins going to the server. For a GOLD FARMER, that's what I want. I liquidate all the assets on your account, and then sell the gold to some sucker with too much money. EVE had that issue for a while, so does WoW. Mind you, their servers weren't hacked (and their design matches what I described btw), but instead they stole the credentials using viruses or spoofed websites.

Edited by Washu, 03 August 2013 - 04:11 PM.

In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.
ScapeCode - Blog | SlimDX


#20 Washu   Senior Moderators   -  Reputation: 4486

Like
2Likes
Like

Posted 03 August 2013 - 07:42 PM

Let me try and explain this a bit better:

If you have a set of game servers that are all exposed to the internet (port forward, just behind a firewall, on the DMZ, whatever). These servers, obviously, have to talk to the database. So clearly there's an open connection between them and the database. Even if the database is behind another firewall, on a different network segment, or any number of other network design decisions.

But let us take the example of a set of port forwards to the different game servers, with the database server on the same network as the game servers.

If all of the game servers are on the same network then, once one is compromised, then it becomes fairly trivial to move laterally to all the other machines *on that segment of the network*. I.e. if I compromise the server at port 6000, I have control over that machine. I can now establish LOCAL connections from that machine to all of the other machines on that network segment, INCLUDING the database server.

The chances of you detecting and stopping such an intrusion in time to prevent damage, or the theft of data, is rather minuscule.

Once someone is ON your servers, no amount of encryption will help you. They can poke directly into memory to yank the keys, passwords, algorithms, or anything else they want out. They can decrypt your configuration files and database connection strings with minimal effort. Frankly, once they're on that segment of the network, that segment is compromised and NOTHING on it is guaranteed.

Now, if we change the network layout slightly, we end up improving security immensely. Specifically, if we move the database server off the local network and onto an internal LAN, then setup a firewall between that LAN and the DMZ that the game servers occupy we have introduced an additional barrier. No longer can I simply remote straight into the database server and say, make a copy of the raw database file for later examination. I'm reliant on connecting via the game servers into the database and running queries. Those queries will only have the permissions of the user that is used by the game servers, although those permissions are usually pretty wide since the game servers have to write changes to the database. Nevertheless, this introduces an additional step, and increases the chances of my detection significantly. Any serious queries I write will introduce a noticeable database load, and it makes it a lot harder for me to simply disguise my hacking as random crap that's getting thrown at the server. While this is good, its not great. The chances of detection are still really low, and I can still end up doing bulk queries and getting large amounts of data out of the system before you ever notice.

Now, if we add an intermediary between the client and the game servers we can impose additional layers of security. In addition we get additional scalability options. Firstly, if we have a DMZ with a series of proxy servers on it, with those proxy servers doing nothing but translating client packets into game events, which are send to the game servers, then we can scale the networking side of things (provided we have a decent backbone between the game servers and the proxies), fairly easily. Too much traffic for one proxy to handle? Toss up another and tell the login server about it.

The game servers are then put behind the firewall with the database. The only way in or out of the local area network is via communication with the game servers. The database is not exposed to the proxy servers at all. Since the game servers only communicate with the proxy servers via game events (i.e. internal packets), then the only ways to attack the game servers, and thus gain direct access to the database, would be to work out YET ANOTHER EXPLOIT and inject that into one of the game servers. In addition, since the game servers only respond to game events, I cannot issue bulk queries and obtain copies of your database WITHOUT hacking my way through that barrier.

Now, if your proxies are doing the validation and translation of packets, batching up zone events and sending them to the correct game server, etc. Then your game servers should be doing validation of those events, such as running physics, etc. This does not exempt the game servers from running basic validation, such as ensuring packet format and length are correct for the event type specified.

This significantly increases the chances of detection. Something as simple as trying a buffer overflow on say a player position update request would likely be caught. Too many such requests might ban that account, but if you kept seeing those requests in the admin log, you would probably open up your sniffer and see whose running a bot.

Now, its not perfect. Someone can STILL get in, and you can still not notice them. But the barrier to entry has been significantly increases with relatively little cost. For any REAL MMO you're going to have to have multiple servers running a single shard ANYWAYS.

As a reference, here is the architecture of the EVE node system, the VPN links are firewalled (i.e. DMZ):

e2lm.png

(This is from their GDC 2009 presentation)


Edited by Washu, 03 August 2013 - 07:43 PM.

In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.
ScapeCode - Blog | SlimDX





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS