Online Server Architecture
Author
So, I've been fiddling around with network programming for a while now and I've been working on an online game engine. The engine is currently designed to serve games of 32 players; however, I would like to expand this to be able to serve at least a few hundred players.
Before I get into some server architecture issues, I thought I'd bring up some other issues to which someone may be able to give some helpful advice.
This is more of an online shooter engine than an online rpg engine, so naturally I'm fighting a loosing battle with bandwidth. I havn't been able to test the engine extensively, but from what I can calculate a regular broadband connection can only serve 32-64 players.
A normal tcp message has a header length of 28 bytes, so in order to save bandwidth I try to pack as many messages into a single tcp message as possible. The rooms are quite small (at least for right now) so the average package contains a movement update for nearly every player in the room, plus messages to create several bullets (out going at least). Each message contains the standard TCP header and then an unsigned short for the length of the entire message. Several smaller messages are put into each message. Each of the smaller messages has an unsigned short to identify which type of message it is and an unsigned short for the length of the message.
I currently update the player movement 10 times a second. I was originally trying to get away with only updating 6 times a second, but the movement seemed way too choppy. The game is 2d so each movement message takes 6 bytes for each character within viewing distance. The average for this message alone is (20*6*10 + 34)*32. By my calculations that equals 39488 Bps or roughly 39 KBps, which is within a standard broadband connection's upload capacity.
I don't have much knowledge of the connections which commercial game servers use; however, I am fairly certain that some illegal WoW private servers are running on DSL/Cable. I also know that some of them are using T1 and T3 connections; however, it frustrates me that WoW, which, in my opinion, should use much more bandwidth, is able to use less bandwidth than my server.
I was just wondering about what is a logical number of players that could play on a standard broadband connection? I thought 32-64 was logical, but these illegal WoW servers are making me wonder if 128 is possible.
Well, now that I've spent a great deal of time on that, I figure I'll get to the point.
This is one of those projects that started off as: "I'll make this as simple as possible" and is turning into: "Well what about x, y and z?" Originally I was just going to make a server that didn't even use SQL could serve 16 players and that was about it. Slowly I started making things a little more complicated because it seemed like it wouldn't be much more work. Now I'm to the point where I've already included a MySQL database, separate web server, separate login server, and then an actual game server. I'm trying to figure out a few things about that and also how best to set up the architecture.
I read that Warhammer Online has an architecture somewhat similar; however, that was from a unreliable source and I feel that they would have servers just for caching information from the MySQL server and probably several other servers to do other things such as instancing. However, this engine isn't going to come with all of the bells and whistles of instances and is not going to be massive, so I think I can suffice with the four separate servers I have mentioned. What do you think?
Additionally, I'm wondering about the MySQL server a lot more than I thought I would. MySQL seems to be fairly fast, and in an online shooter it isn't used a whole lot. However, I consistently read that Database servers are the biggest problem in online games. I currently have 0 caching, but I really don't do very many queries. I update the database at the end of a game to let it know what happened, I query here and there before the game starts, and I query at the login -- that's about it (for now at least). I havn't load tested the server at all, but I can't imagine that being a problem. However, I want the game to be scalable, so I'm just wondering if I need to address this or not. What do you think?
Furthermore, I'm starting to worry about authentication from the login server to the game servers. I spent a lot of time working on a moderately secure login server and I would like the authentication to the game servers to be equally secure. Currently, I'm using what I think is private key security. When the game server joins the serverlist server, the serverlist/login server sends a private key to the game server and stores that. When a client sends a request to join a server from the login server, the login server generates a public key concatenates that to the private key and sends the hash of that and the public key to the client. The client sends this public key and the hash to the game server. If the game server gets the hash you sent it, then it is satisfied. That seems pretty secure to me, but I'm no expert. What do you think?
Additionally, I'm wondering about the game server. It isn't designed to be a massive world. It is designed to be a small room that contains a small number of players (My goal is to be able to put 4 teams of 32 or 128 players in one room). So the server is extremely different from current MMORPGs.
The engine will be open source when it is released. I'm planning to target developers that want to quickly make an online game of this sort and are willing to let players (more or less) host game servers. The login/database server and obviously the web server will need to be managed by the developer. But it is important for me to be able to have the games served on a standard broadband connection (I can't stress that enough). If you can make any educated estimate as to how many players you think is plausible, I would appreciate that.
Finally, I made a topic a few weeks ago about scripting. About the time I made this post is when I started making all these changes to my plans. I want to be able to have a core server that does all the extremely basic things all these types of games will need to do. From that I want to have an ability to take messages from the server (C++) and work with them in LUA. Originally I was going to write it so that the server did this internally, but that became too much of a burden on me (I don't know LUA or anything about using it with C++). So, instead, I just made it very simple to turn the server into a DLL that I figure anyone wanting to do this in LUA will be able to figure out how to do themselves (perhaps I will be able to help when I have some time). Anyways, now that I'm starting to aim for bigger goals I'm wondering if this is still logical? The way I do this is fairly simple. Anything the server recognizes it processes and anything it doesn't it pushes onto a stack that an external program can pop off to process itself. I think that since the shard/game server really hasn't gotten any bigger expectations that this is still very logical. I just thought that I'd bring it up again to a different audience that might have a different opinion.
Sorry for the large post. Sorry for rambling on a lot. Thank you for reading. I will greatly appreciate any input. Please don't flame this like a standard MMO thread. I feel this is a little more meaningful than: "I want to make an MMO but I don't know anything about anything." I think I tried a few things and I know at least a little bit about the topic and I'm not even asking about MMOs.
I didn't read the whole thing (didn't have enough time to) but I can reply to some of it. First off, TCP is typically 20 bytes I believe, not 28. But you also have the IPv4 header which is another 20 bytes. You also mentioned you have a ushort on your messages, so thats 42 bytes overhead before even getting to the information.
Shooters typically use a completely different networking scheme then MMORPGs. A shooter (at least the typical kind) demands very low latency and a lot of changing entities. A MMORPG, on the other hand, is typically much slower and automated. Your character may be smacking away at that dragon, but if its one of those many MMORPGs that have automated targeting, all the server really needs to do is say, "A started attacking B" and send the occasional stat/damage notifications to those who are interested (which is usually just the one fighting). Also, since fights are often much slower and the enemy is usually a NPC, latency is nowhere near as critical.
So how do they get away with so little bandwidth? Well, for one, a FPS usually works by sending the state of nearby entities constantly, using UDP to do this. No data is retransmitted, and its not uncommon to receive an update for something that hasn't changed, but it results in very low latency. TCP-based games, on the other hand, know that their data will eventually reach the target, so they can send updates just when something changes. So now you got a more event-based, slower scheme using TCP and a constantly-sending, "shotgun"-update scheme with UDP, which is going to have much lower latencies at a much higher bandwidth cost. And because of these hugely different schemes is why it can be so hard to merge the two.
Whoop, gotta run!
Shooters typically use a completely different networking scheme then MMORPGs. A shooter (at least the typical kind) demands very low latency and a lot of changing entities. A MMORPG, on the other hand, is typically much slower and automated. Your character may be smacking away at that dragon, but if its one of those many MMORPGs that have automated targeting, all the server really needs to do is say, "A started attacking B" and send the occasional stat/damage notifications to those who are interested (which is usually just the one fighting). Also, since fights are often much slower and the enemy is usually a NPC, latency is nowhere near as critical.
So how do they get away with so little bandwidth? Well, for one, a FPS usually works by sending the state of nearby entities constantly, using UDP to do this. No data is retransmitted, and its not uncommon to receive an update for something that hasn't changed, but it results in very low latency. TCP-based games, on the other hand, know that their data will eventually reach the target, so they can send updates just when something changes. So now you got a more event-based, slower scheme using TCP and a constantly-sending, "shotgun"-update scheme with UDP, which is going to have much lower latencies at a much higher bandwidth cost. And because of these hugely different schemes is why it can be so hard to merge the two.
Whoop, gotta run!
christ, lots of questions...
As Spodi says, the choice between TCP / UDP, is in a way, similar to low bandwidth / low latency.
MMORPGs are event-based, require little bandwidth, a reliable transmission, and not for accurate, fast-paced action. A bit of latency, although annoying, won't really damage the game mechanics.
FPS shooters are the opposite, they need to support fast world updates, instant input response and minimum latency.
That's why a FPS mmorg would be pretty hard to code to satisfy both requirements.
There is a third model, used for RTS games, that use a deterministic engine, but you don't wanna go there.
For the player limit for broadband servers, you can work it out. Calculate the average packet size for a player/entity update, multiply by the average network rate, multiply by number of connections, multiplied by the number of 'visible' players/entities seen by that connection. So it will grow very fast as your connection count grows (as it will add more players and entities like grenades, rockets, stuff spawned by players, into the potentially visible set for each connections).
Typically, 16-24 is an acceptable limit. 64 would be really pushing it and would require a high spec broadband connection. Remember that it is upstream performance you are worried about, and it is very poor on Mr Anonymous' DSL.
That's why console games with no dedicated servers are more peer-to-peer, to spread the load among all the clients. And then again, it is still a challenge to support more than 16 players on a broadband connection. Having automated load-balancing is a must to keep the game (relatively) flowing smoothly.
As for authentication, I'm no expert but the way you describe is pretty much how game sessions work. You advertise your game session on a master (match making) server, that gives you a key and unique session ID, and anyone wanting to connect to your server has to get the key off the 'matchmaking' server. Then all the packets are encrypted using that key. So each packet will also have an encryption header, that will increase your packet size (by quite a lot).
The authentication can be three-way, if you have a separate, dedicated login server as well.
Consoles provide that functionality for you, but finding equivalent support for PCs is much more difficult (and costly). You'd be looking at gamespy, demonware, ect... Implementing it yourself is a huge task too, but feasable noneless.
Also, do not forget NAT punch-through, a matchmaking server can help you with that (it does in a way, providing you with the session host's internet address).
Thirdly, if you plan on having broadband users hosting, it can be useful to support host migration, but it is very hard to implement.
Oh, and for a SQL database, this is usually redundant for FPS (fast paced) games. You can use a database for registering game sessions, player stats, data mining, ect... but it is overkill for in-game data. The pace of the game is far too high to be querying servers for entity states. You want minimum latency and the data is non-persistent.
As Spodi says, the choice between TCP / UDP, is in a way, similar to low bandwidth / low latency.
MMORPGs are event-based, require little bandwidth, a reliable transmission, and not for accurate, fast-paced action. A bit of latency, although annoying, won't really damage the game mechanics.
FPS shooters are the opposite, they need to support fast world updates, instant input response and minimum latency.
That's why a FPS mmorg would be pretty hard to code to satisfy both requirements.
There is a third model, used for RTS games, that use a deterministic engine, but you don't wanna go there.
For the player limit for broadband servers, you can work it out. Calculate the average packet size for a player/entity update, multiply by the average network rate, multiply by number of connections, multiplied by the number of 'visible' players/entities seen by that connection. So it will grow very fast as your connection count grows (as it will add more players and entities like grenades, rockets, stuff spawned by players, into the potentially visible set for each connections).
Typically, 16-24 is an acceptable limit. 64 would be really pushing it and would require a high spec broadband connection. Remember that it is upstream performance you are worried about, and it is very poor on Mr Anonymous' DSL.
That's why console games with no dedicated servers are more peer-to-peer, to spread the load among all the clients. And then again, it is still a challenge to support more than 16 players on a broadband connection. Having automated load-balancing is a must to keep the game (relatively) flowing smoothly.
As for authentication, I'm no expert but the way you describe is pretty much how game sessions work. You advertise your game session on a master (match making) server, that gives you a key and unique session ID, and anyone wanting to connect to your server has to get the key off the 'matchmaking' server. Then all the packets are encrypted using that key. So each packet will also have an encryption header, that will increase your packet size (by quite a lot).
The authentication can be three-way, if you have a separate, dedicated login server as well.
Consoles provide that functionality for you, but finding equivalent support for PCs is much more difficult (and costly). You'd be looking at gamespy, demonware, ect... Implementing it yourself is a huge task too, but feasable noneless.
Also, do not forget NAT punch-through, a matchmaking server can help you with that (it does in a way, providing you with the session host's internet address).
Thirdly, if you plan on having broadband users hosting, it can be useful to support host migration, but it is very hard to implement.
Oh, and for a SQL database, this is usually redundant for FPS (fast paced) games. You can use a database for registering game sessions, player stats, data mining, ect... but it is overkill for in-game data. The pace of the game is far too high to be querying servers for entity states. You want minimum latency and the data is non-persistent.
Quote:I am fairly certain that some illegal WoW private servers are running on DSL/CableSure. They could also run on 28.8k modem. That doesn't mean they scale or that they are playable. This type of MMORPGs also limits the bandwidth to 3 kilobytes per sec (not 30) peak, and 1k average.
Quote:I was just wondering about what is a logical number of players that could play on a standard broadband connection?That depends on the logic you're using. 1000, even 10,000 is not impossible. It just requires you to adopt a different simulation model.
Quote:and in an online shooter it isn't used a whole lotBetter yet, it's not used at all, since there's no persistent world. It would be used for either leaderboards, or account information or such.
Quote:so I think I can suffice with the four separate servers I have mentioned
Quote:Now I'm to the point where I've already included a MySQL database, separate web server, separate login server, and then an actual game server.
What exactly did you gain with this, apart from configuration nightmare.
- Web server: bandwidth heavy, requires complex administration
- SQL server: memory/disk heavy, puts load on network and CPU
- Login server: will be sitting idly (128 logins are handled in around a second)
- Game server: still overloaded
What would make sense, is not having a "game server", but instead having "instance server". So for each room you spawn a new instance. Then, the login server takes care of managing them, and assigning players to them.
You also drop the SQL server, and use a proxy to access that. This gives you the benefit of either using embedded SQL, or keeping all data cached. You mentioned your entire state isn't too large. You can even drop the SQL, since it doesn't sound like you need it.
Quote:Additionally, I'm wondering about the game server. It isn't designed to be a massive world. It is designed to be a small room that contains a small number of players (My goal is to be able to put 4 teams of 32 or 128 players in one room).
This design could end up causing problems as you'll have an n-squared bandwidth increase if all players are within 'relevancy' distance of each other. Most games with large numbers of players are usually designed to keep players in separate smallish groups, you can see the problems that happen when large numbers of players congregate in WoW.
(If you have 32 players all together you need to send 32 updates to each player, so that's 32*32=1024. If you have 2 separate groups of 16 players on the same server but in different areas/rooms/whatever that's 16*16*2=512. Now imagine how that scales if you can design your game so you can have 1000 players on a server but all in smallish groups, compared to 1000 players all in the same room (1million updates!).)
First, as has been said, the protocol overhead for a TCP packet is at least 40 bytes (more if certain options are enabled). UDP is 28.
Second, if you have 30 users who are all right next to each other, and you want twitch update rates, and you have limited upload speed, then something has to give. Either reduce the number of players, or increase the bandwidth, or remove the twitch requirement.
WoW is an example of the latter: it does not give twitch response to movement at all. If the link is heavily used, you may get one update a second or less for players you can see (and no updates for players you can't see).
When it comes to bandwidth, the larger systems typically are hosted in well connected co-location facilities, where the physical link isn't much of a problem. Instead, you "commit" to a certain average throughput, and if your average goes over that, there's usually a surcharge. Typical commit prices are in the $5-10/Mbit/month range, so a Gbit/s commit is about $5k-$10k per month (very roughly -- each deal is different at these levels).
Second, if you have 30 users who are all right next to each other, and you want twitch update rates, and you have limited upload speed, then something has to give. Either reduce the number of players, or increase the bandwidth, or remove the twitch requirement.
WoW is an example of the latter: it does not give twitch response to movement at all. If the link is heavily used, you may get one update a second or less for players you can see (and no updates for players you can't see).
When it comes to bandwidth, the larger systems typically are hosted in well connected co-location facilities, where the physical link isn't much of a problem. Instead, you "commit" to a certain average throughput, and if your average goes over that, there's usually a surcharge. Typical commit prices are in the $5-10/Mbit/month range, so a Gbit/s commit is about $5k-$10k per month (very roughly -- each deal is different at these levels).
Author
I'm well aware of the n2; however, I'm well aware of the saying: "The more the merrier." Let's face it, a game with 32 players is more fun than a game with 16 players. 16 players is plenty, but I really want to get at least 32 in a room.
So, I guess I was right when I said that 32 is pushing the max of a broadband connection.
The MySQL server is staying. I need it because this game is a mix between an rpg and a shooter (it has rpg rules, but ultimately plays like a shooter). The database is still very important; however, as I said, the database is not really even used during game play. The shards make queries before the game starts and after it is over and the login server makes queries to authenticate users.
The shard server is a single room -- I think my earlier post did not make this clear. However, the master server does not spawn these -- there is a set number.
The servers are split programs, they obviously can still be run on one computer and the split makes the difference in speed minimal; however, when the game becomes much more popular the split is the difference in the world. However, I can see where you all are coming from when you say it seems a bit pointless. I honestly don't expect anything good enough to need this architecture to come from this engine; however, I'm programming it this way because I'm optimistic and it is helping me learn how these types of applications work.
Furthermore, I want to stay with TCP rather than using UDP. UDP does have a header size that is 20 bytes smaller; however, 20*10*32 is only 6KBps. That could serve a few more players, but it's not going to put 32 more players in a room. The problem is the n2 movement of the players, which there isn't really a work around for, other then simple having less players. Additionally, I don't know if it is just me, but the UDP programs I have created in the past have had problems with players routers rendering them unable to play. The one thing I don't want is another reason for someone not to play my game. They already have WoW, Halo, GTAIV, etc... I don't need any help =p
Thank you all for your input. I would appreciate anything else you have to add, especially about the authentication and server architecture.
edit:
Bah.. Hplus you are a genius that is a trick I used my first online game that I had completely forgotten about. I'm wondering exactly how low I can make this in an online shooter, though. I'm thinking if I put a cap on the amount of data a player can send per second this could fix a lot of things. 48*10 is the average minimum and roughly 32 KBps is what I'm targeting for server bandwidth. I'm thinking that I could allow a max of 2KBps to 32 players. Obviously that is a great deal more than the server can deal with, but I'll have other precautions in place to deal with that.
I understand I'll have to work all of this out myself until everything looks right. It is very hard for someone to arbitrarily throw numbers out there, but if you have anything to add, I would appreciate it.
Once again, thank you all.
So, I guess I was right when I said that 32 is pushing the max of a broadband connection.
The MySQL server is staying. I need it because this game is a mix between an rpg and a shooter (it has rpg rules, but ultimately plays like a shooter). The database is still very important; however, as I said, the database is not really even used during game play. The shards make queries before the game starts and after it is over and the login server makes queries to authenticate users.
Quote:What exactly did you gain with this, apart from configuration nightmare.
- Web server: bandwidth heavy, requires complex administration
- SQL server: memory/disk heavy, puts load on network and CPU
- Login server: will be sitting idly (128 logins are handled in around a second)
- Game server: still overloaded
The shard server is a single room -- I think my earlier post did not make this clear. However, the master server does not spawn these -- there is a set number.
The servers are split programs, they obviously can still be run on one computer and the split makes the difference in speed minimal; however, when the game becomes much more popular the split is the difference in the world. However, I can see where you all are coming from when you say it seems a bit pointless. I honestly don't expect anything good enough to need this architecture to come from this engine; however, I'm programming it this way because I'm optimistic and it is helping me learn how these types of applications work.
Furthermore, I want to stay with TCP rather than using UDP. UDP does have a header size that is 20 bytes smaller; however, 20*10*32 is only 6KBps. That could serve a few more players, but it's not going to put 32 more players in a room. The problem is the n2 movement of the players, which there isn't really a work around for, other then simple having less players. Additionally, I don't know if it is just me, but the UDP programs I have created in the past have had problems with players routers rendering them unable to play. The one thing I don't want is another reason for someone not to play my game. They already have WoW, Halo, GTAIV, etc... I don't need any help =p
Thank you all for your input. I would appreciate anything else you have to add, especially about the authentication and server architecture.
edit:
Quote:Second, if you have 30 users who are all right next to each other, and you want twitch update rates, and you have limited upload speed, then something has to give. Either reduce the number of players, or increase the bandwidth, or remove the twitch requirement.
WoW is an example of the latter: it does not give twitch response to movement at all. If the link is heavily used, you may get one update a second or less for players you can see (and no updates for players you can't see).
Bah.. Hplus you are a genius that is a trick I used my first online game that I had completely forgotten about. I'm wondering exactly how low I can make this in an online shooter, though. I'm thinking if I put a cap on the amount of data a player can send per second this could fix a lot of things. 48*10 is the average minimum and roughly 32 KBps is what I'm targeting for server bandwidth. I'm thinking that I could allow a max of 2KBps to 32 players. Obviously that is a great deal more than the server can deal with, but I'll have other precautions in place to deal with that.
I understand I'll have to work all of this out myself until everything looks right. It is very hard for someone to arbitrarily throw numbers out there, but if you have anything to add, I would appreciate it.
Once again, thank you all.
if you play counterstrike source, you can see the adaptive update working when you spectate. You'll see players on the minimap moving at different rates, depending on their visibility from your camera viewpoint. from 20 fps down to a few updates a second (maybe 2 or three). It is a must if you want to minimise bandwidth in a very busy game.
Secondly, have you thought of using Peer to Peer to reduce the server load? I know peer to peer isn't as secured as a pure server clients, but if you really want that amount of players around on DLS / cable lines, you should consider letting the players do some work themselves. They'll have bandwidth to spare (voice comms is a given).
128 players, updating at 20 fps in a shooter environment is a LOT of data. And anything below 10 fps will feel laggy and unresponsive.
I'm not sure the internet is mature enough for that sort of gaming requirements. You can prototype on LAN, and that will give you an idea of the load for the servers and clients respectively. Then you may have to think of downscaling a bit. The worse case scenario is pretty scary, to the point where the server could completly overload.
In any case, it's a design decision. If you spec your requirements and the constraints of a real world scenario, you'll see if both can fit (which I doubt very much).
That, or the gameplay will have to be something more similar to an RPG, :/
Secondly, have you thought of using Peer to Peer to reduce the server load? I know peer to peer isn't as secured as a pure server clients, but if you really want that amount of players around on DLS / cable lines, you should consider letting the players do some work themselves. They'll have bandwidth to spare (voice comms is a given).
128 players, updating at 20 fps in a shooter environment is a LOT of data. And anything below 10 fps will feel laggy and unresponsive.
I'm not sure the internet is mature enough for that sort of gaming requirements. You can prototype on LAN, and that will give you an idea of the load for the servers and clients respectively. Then you may have to think of downscaling a bit. The worse case scenario is pretty scary, to the point where the server could completly overload.
In any case, it's a design decision. If you spec your requirements and the constraints of a real world scenario, you'll see if both can fit (which I doubt very much).
That, or the gameplay will have to be something more similar to an RPG, :/
On a related note, I was recently (a few weeks ago) told about a MMORPG that was implementing FPS elements such as manual aiming with the bow and such, but can't remember the name of it. And no, it wasn't Hellgate:London. When I went to look at it, it seemed that they actually dropped the manual aiming since it was just way too problematic, and now you do the typical "click here to auto-shoot!".
The one game I can think of that has actually done something like this is Hellgate:London, but even still thats just heavily instanced which I don't consider being in the same world. If you go to location X, you should see everyone else in the world who is at location X, not just 2 other people who went there at the same time as you.
My point? You're up for a huge challenge by trying to mix the two. Its something everyone seems to want to do, but even with the pros the results are less than desirable. But in any case, I don't think you'll be able to put 128 active players in one room on a home connection. Assuming you're even using an event-based update system, if one person moves, thats 40*128 bytes of overhead on TCP/IPv4. Factor in the message length header (2 bytes), the message (lets say about 6 bytes), and you still got yourself (40+2+6)*128 or 6144 bytes for that one movement. Even backed by the most uber of connections, its still going to take some amazing hardware and techniques to get 128 players updates fast.
...even if my math is off somewhere, which I wouldn't be surprised if it is (what can I say, its early), the point remains that it will be difficult^2 to keep 128 players updated about 127 other players. ;)
The one game I can think of that has actually done something like this is Hellgate:London, but even still thats just heavily instanced which I don't consider being in the same world. If you go to location X, you should see everyone else in the world who is at location X, not just 2 other people who went there at the same time as you.
My point? You're up for a huge challenge by trying to mix the two. Its something everyone seems to want to do, but even with the pros the results are less than desirable. But in any case, I don't think you'll be able to put 128 active players in one room on a home connection. Assuming you're even using an event-based update system, if one person moves, thats 40*128 bytes of overhead on TCP/IPv4. Factor in the message length header (2 bytes), the message (lets say about 6 bytes), and you still got yourself (40+2+6)*128 or 6144 bytes for that one movement. Even backed by the most uber of connections, its still going to take some amazing hardware and techniques to get 128 players updates fast.
...even if my math is off somewhere, which I wouldn't be surprised if it is (what can I say, its early), the point remains that it will be difficult^2 to keep 128 players updated about 127 other players. ;)
Author
I'm trying to think of all the ways to reduce this load as much as possible. I've come up with a few ideas in short amount of time and I'm sure some of you can come up with some even better ones.
First, as Hplus mentioned, I'm going to put a limit on the amount of bandwidth each player can hog. I'm also going to update players farther away less frequently than players close in range. I think this is fairly standard practice.
Second, originally I was going to make the rooms fairly small; however, the smaller the area the better the chance that more players are in viewing distance. So, I decided that I'm going to change the game play a little bit and make the rooms a great deal larger. I'm also going to give players incentives to not be in large groups.
For example, the main play mode is a base control game. Players get points for controlling bases for a certain amount of time and after a team reaches a certain amount of time, the team wins.
If I spread these bases out a little bit and add more of them, players quickly become spread fairly thin. So, instead of having average 8 on 8 to 16 on 16 encounters, players will be having 3 on 3 to 6 on 6 encounters.
I feel that this could reduce the average player packet from having to send 20-30 players to having to send 6-12 players. I realize that that reduction isn't going to solve anything by itself, though.
Finally, I'm considering adding a peer-to-peer part to the game where p2p messages are sent through UDP. I was opposed to p2p and UDP entirely running the system, but I think that if I use the p2p UDP to only control player movements to their own team that everything will work out.
I'm concerned if this will be able to be secure, though. I'm planning to let players send messages to players on their team to update their movement. If the want to mod their messages and send false coordinates to their own team, that really isn't cheating, since I don't see how that is giving them an advantage. It seems that I could fairly simply be able to keep players from interfering with the server messages to mod their movement on the enemies screens, but I'm wondering if I can do this as securely as I think.
I planning to do this fairly simply, using a private and public key system. The Shard server will send players the IPs they will need to send to and they will be allowed to receive from. Is there a way these messages could be hijacked by players from the other team to allow them to send messages to overwrite the server messages for their movement?
I'm still planning to have the server control the movement from opposing teams. But this will reduce the load of the server by a third easily; however, I think this could possibly stress 56K players. They will be receiving tons of UDP messages rather than just one. So, even with only 32 players, 15 players could possibly be sending a connection 28*20*15. That is only 8KBps, if I calculated correctly, which I probably didn't. However, the scalability becomes more of an issue this way I think. If you expand that to 128 players, which is really my biggest target, that leaves 63 players that could possibly be sending 20 messages a second. That becomes 31KBps, which has to go both ways, plus the additional load from the server. That becomes something that could possibly clog up a crappy DSL line and wondering if you could get 256 players in a room becomes a joke.
So, I thought that I could modify that system a little so that only a few players per team would be sending the messages and the server would only be sending a few players per team messages. The those players must turn around and disperse those packages to their team. This sounds like a very interesting idea to me; however, it seems like if it were as good of an idea as I think, someone else would be doing this and I don't think anyone is doing this.
Furthermore, I'm unaware of a way to easily find the fastest connections and I'm wondering what happens when one of the "important players" loses their connection from the server. My intention was never to create something for commercial games (I'm not that good so it would be stupid to even try), but when I start talking about rooms of 128 players that sort of sounds like a commercial game. Am I just getting ahead of myself? Is it stupid to try and put such small burden on servers and put such high burden on clients? Should I just stop caring about hosting on a broadband connection if I want to put 128 players in a room? I know I could do the coding for what I have described, I just don't know if it is logical and if it will be secure.
So, is this goal illogical/unrealistic? If I go back to the 32 player target, will this goal be very difficult to achieve? I tested a while back with a different server on a faster connection with only 16 players and the server handled it fine. However, the strain on the server grows exponentially with the number of players added.
Once again, you all have been extremely helpful and I really appreciate everything so far. If you have any ideas (even if they are a totally different approach), I would really appreciate it if you would share them. Thank you.
First, as Hplus mentioned, I'm going to put a limit on the amount of bandwidth each player can hog. I'm also going to update players farther away less frequently than players close in range. I think this is fairly standard practice.
Second, originally I was going to make the rooms fairly small; however, the smaller the area the better the chance that more players are in viewing distance. So, I decided that I'm going to change the game play a little bit and make the rooms a great deal larger. I'm also going to give players incentives to not be in large groups.
For example, the main play mode is a base control game. Players get points for controlling bases for a certain amount of time and after a team reaches a certain amount of time, the team wins.
If I spread these bases out a little bit and add more of them, players quickly become spread fairly thin. So, instead of having average 8 on 8 to 16 on 16 encounters, players will be having 3 on 3 to 6 on 6 encounters.
I feel that this could reduce the average player packet from having to send 20-30 players to having to send 6-12 players. I realize that that reduction isn't going to solve anything by itself, though.
Finally, I'm considering adding a peer-to-peer part to the game where p2p messages are sent through UDP. I was opposed to p2p and UDP entirely running the system, but I think that if I use the p2p UDP to only control player movements to their own team that everything will work out.
I'm concerned if this will be able to be secure, though. I'm planning to let players send messages to players on their team to update their movement. If the want to mod their messages and send false coordinates to their own team, that really isn't cheating, since I don't see how that is giving them an advantage. It seems that I could fairly simply be able to keep players from interfering with the server messages to mod their movement on the enemies screens, but I'm wondering if I can do this as securely as I think.
I planning to do this fairly simply, using a private and public key system. The Shard server will send players the IPs they will need to send to and they will be allowed to receive from. Is there a way these messages could be hijacked by players from the other team to allow them to send messages to overwrite the server messages for their movement?
I'm still planning to have the server control the movement from opposing teams. But this will reduce the load of the server by a third easily; however, I think this could possibly stress 56K players. They will be receiving tons of UDP messages rather than just one. So, even with only 32 players, 15 players could possibly be sending a connection 28*20*15. That is only 8KBps, if I calculated correctly, which I probably didn't. However, the scalability becomes more of an issue this way I think. If you expand that to 128 players, which is really my biggest target, that leaves 63 players that could possibly be sending 20 messages a second. That becomes 31KBps, which has to go both ways, plus the additional load from the server. That becomes something that could possibly clog up a crappy DSL line and wondering if you could get 256 players in a room becomes a joke.
So, I thought that I could modify that system a little so that only a few players per team would be sending the messages and the server would only be sending a few players per team messages. The those players must turn around and disperse those packages to their team. This sounds like a very interesting idea to me; however, it seems like if it were as good of an idea as I think, someone else would be doing this and I don't think anyone is doing this.
Furthermore, I'm unaware of a way to easily find the fastest connections and I'm wondering what happens when one of the "important players" loses their connection from the server. My intention was never to create something for commercial games (I'm not that good so it would be stupid to even try), but when I start talking about rooms of 128 players that sort of sounds like a commercial game. Am I just getting ahead of myself? Is it stupid to try and put such small burden on servers and put such high burden on clients? Should I just stop caring about hosting on a broadband connection if I want to put 128 players in a room? I know I could do the coding for what I have described, I just don't know if it is logical and if it will be secure.
So, is this goal illogical/unrealistic? If I go back to the 32 player target, will this goal be very difficult to achieve? I tested a while back with a different server on a faster connection with only 16 players and the server handled it fine. However, the strain on the server grows exponentially with the number of players added.
Once again, you all have been extremely helpful and I really appreciate everything so far. If you have any ideas (even if they are a totally different approach), I would really appreciate it if you would share them. Thank you.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
