I am wondering if what I have developed so far is a good way to approach designing a two dimensional side scrolling massively multi-player server (2000+ clients)
I have programmed and designed a "prototype" to see whether the following solution would work well:
1) Completely programmed in C#
2) Uses NoDelay TCP using .net Sockets (WinSock) therefore I am not using a network library.
3) The server only has two threads in total, one is for accepting TCP connections and another for doing everything else. The other is a high-priority thread in which I may add the task parallel library (TPL) to later on. All this thread does is download traffic, then update the world, then upload all traffic (including any updated world objects from the previous world update) then check the connection status of all connected players, this loops every 16.6667 milliseconds.
4) Data is only sent when someone actually DOES something, i.e. nothing will happen if no one moves, even when someone starts moving left only 1 message would have been sent to all connected players (including the person that moved in the first place) until there is another message indicating this player has stopped (the move key on the keyboard has been released). Both start moving/stop moving messages sent from the server contain positions of where the player started moving from and where the player stopped moving from and the reason for this is so that the server can sync all connected players seamlessly. For example, when the client receives a start moving from here (x,y) the client program is then on its own and updates the player's position each frame starting from the (x,y) position that was received in the message. The same thing happens when the client receives a stop moving from here (x,y), in which the client starts moving the velocity to 0 of the specified player that was moving in the first place.
5) The whole game is server-based, players have no authority at all (AT ALL). When a player wants to move the input gets sent to the server in which the server transmits back where the player should start moving from. Same thing happens for a client to stop moving.
6) I am designing two programs and one library, which includes: the client, the server and the library that both client and server use. The client code and server code will probably 50% similar, the reason why I am doing this is so I can design the server as efficiently as possible.
Result: IT WORKS (haven't tested it large scale yet though)! The only problem I am seeing though is that it is completely server based, if a player that has 300 latency presses the key to move right, they would only move right after 300 milliseconds PLUS the frame time (16.66667). I'm thinking of making the client's player be client-based so they move instantly but if this is the case I will need to make sure the client can send a "I STARTED MOVING LEFT FROM HERE" message and make sure the server checks if the player is not x-units away from the last received position, if it is though the server has authority and moves the player back to the right position.
The question: Id would just like to know what other programmers thing about this approach. I probably wrote this awfully but I am usually terrible at writing what my own code does, sorry :|
All replies are greatly appreciated.
Thanks, Xanather.
Massively Multiplayer Server Design
The FAQ has several links to ways of thinking about this, including the "Gaffer" networked character physics article, the "Source" networking/lag compensation article, and the "Quake III" article by Brian Hook.
In general, if you let the player move ahead on the local client, you may end up with situations where, on machine 1, player 1 is first to a particular goal, but on machine 2, player 2 is first to a particular goal, and you'll have to figure out how to hide this discrepancy and let the server decide.
In general, if you let the player move ahead on the local client, you may end up with situations where, on machine 1, player 1 is first to a particular goal, but on machine 2, player 2 is first to a particular goal, and you'll have to figure out how to hide this discrepancy and let the server decide.
Author
Ok I will look at the FAQ thank you, the first to a particular goal does not concern me so I suppose I should implement that.
Also, if you're trying to send the updates for 2,000 players, to 2,000 players, that will probably scale pretty poorly. It's important to design the game such that it doesn't generate n-squared interaction situations among all players.
Author
Yes, I'm going to try design it so that each player has a (artificial) screen width of, say for example, (1920 + 200) × (1080 + 200), from the center of the player position (since this is a side scroller) therefore if there are any players outside of these bounds they will not receive any updates of the players that are outside of these bounds, in fact the client computer wont even know someone exists there. Of course though, if there are many people packed into one area there will be a problem you are talking about.
How well do you think this will scale? I honestly have no previous experience in this area.
Messages being sent/received will only be send/received when necessary, and even when messages do get sent (i.e. player used this ability) the server will only send a message size of 9 bytes! in total e.g.
2-byte ushort to indicate the message size
1-byte to represent the type of message (in this example using a ability)
4-byte player id so the receiving client knows who actually used the ability.
2-byte ushort to indicate the type of ability used
total: 9 bytes.
Say if 500 players are connected to one server, and each of the players are in the EXACT same area (which the possibility is extremely low anyway) id see the following:
1. 20 messages per second per client (assuming the maximum - this will probably be much smaller)
2. messages are 12 bytes in size (average - might even be smaller if I'm smart about it)
Results in 20 * 12 * 500 (all messages received from all clients) * 500 (messages relayed by the server back to client) which equals...
58593.75 kilobytes per second (upload and download) for the server.
but only 117.1875 kilobytes per second for each connected client.
Id say that gives me a happy face but I may be completely wrong in either my calculation or overestimating what is actually capable, but I feel strong this is extremely capable for hardware these days.
Another thing to note is that each client has a upload/download buffer of 131072 bytes (too large? i did this so the server can support full 65536 byte message sizes in the upload/download buffers without having to redo the download algorithm and prevent going over the buffer limit).
Do you have any bad thoughts on what I have mentioned?
Replies are appreciated, thanks.
How well do you think this will scale? I honestly have no previous experience in this area.
Messages being sent/received will only be send/received when necessary, and even when messages do get sent (i.e. player used this ability) the server will only send a message size of 9 bytes! in total e.g.
2-byte ushort to indicate the message size
1-byte to represent the type of message (in this example using a ability)
4-byte player id so the receiving client knows who actually used the ability.
2-byte ushort to indicate the type of ability used
total: 9 bytes.
Say if 500 players are connected to one server, and each of the players are in the EXACT same area (which the possibility is extremely low anyway) id see the following:
1. 20 messages per second per client (assuming the maximum - this will probably be much smaller)
2. messages are 12 bytes in size (average - might even be smaller if I'm smart about it)
Results in 20 * 12 * 500 (all messages received from all clients) * 500 (messages relayed by the server back to client) which equals...
58593.75 kilobytes per second (upload and download) for the server.
but only 117.1875 kilobytes per second for each connected client.
Id say that gives me a happy face but I may be completely wrong in either my calculation or overestimating what is actually capable, but I feel strong this is extremely capable for hardware these days.
Another thing to note is that each client has a upload/download buffer of 131072 bytes (too large? i did this so the server can support full 65536 byte message sizes in the upload/download buffers without having to redo the download algorithm and prevent going over the buffer limit).
Do you have any bad thoughts on what I have mentioned?
Replies are appreciated, thanks.
I would expect a large chunk of users on the internet to not be able to reliably sustain receiving 117 kilobytes of payload data per second. I would aim at one-tenth of that, at most.
Also, experience with users shows that they *love* being where other users are. Clumping is, unfortunately, the normal behavior, rather than the uncommon behavior. Thus, game design affordances that can mitigate that would be important. Anything from limited-capacity buildings, to instanced fighting arenas, should be used where appropriate, to reduce number of users in any one single clump.
Finally, 58+ megabytes per second of payload means one gigabit per second of network bandwidth from your server -- it will *not* fit on a 100 Mbit Ethernet port. The cost of a 1 Gbit connection will run something like $4,000-$10,000 per month, depending on specifics. If your 500 players will each pay you $20 per month, that will pay for itself, but I have a sneaking suspicion they won't ;-)
(Then again, the online:offline ratio of players on any game is usually 10:1 or more -- but still, it's a real cost.)
Also, experience with users shows that they *love* being where other users are. Clumping is, unfortunately, the normal behavior, rather than the uncommon behavior. Thus, game design affordances that can mitigate that would be important. Anything from limited-capacity buildings, to instanced fighting arenas, should be used where appropriate, to reduce number of users in any one single clump.
Finally, 58+ megabytes per second of payload means one gigabit per second of network bandwidth from your server -- it will *not* fit on a 100 Mbit Ethernet port. The cost of a 1 Gbit connection will run something like $4,000-$10,000 per month, depending on specifics. If your 500 players will each pay you $20 per month, that will pay for itself, but I have a sneaking suspicion they won't ;-)
(Then again, the online:offline ratio of players on any game is usually 10:1 or more -- but still, it's a real cost.)
I highly suggest you rework your packet-handling code to utilize UDP, as it is much, much, much. much, much. much, much faster (and should be used if programming anything that is meant to receive and send packets quickly).
2) Uses NoDelay TCP using .net Sockets (WinSock) therefore I am not using a network library.
Ehh... This is generally frowned upon, but if you can get away with doing it in your game; sweet!
4) Data is only sent when someone actually DOES something
Why do you multiply by 500 twice? If you think about it, this is what happens (at least, it should) when someone broadcasts a message to other players:
Say if 500 players are connected to one server, and each of the players are in the EXACT same area (which the possibility is extremely low anyway) id see the following:
1. 20 messages per second per client (assuming the maximum - this will probably be much smaller)
2. messages are 12 bytes in size (average - might even be smaller if I'm smart about it)
Results in 20 * 12 * 500 (all messages received from all clients) * 500 (messages relayed by the server back to client) which equals...
58593.75 kilobytes per second (upload and download) for the server.
but only 117.1875 kilobytes per second for each connected client.
Id say that gives me a happy face but I may be completely wrong in either my calculation or overestimating what is actually capable, but I feel strong this is extremely capable for hardware these days.
1) The packet to be sent to the server is composed like so:
00 - Packet-type : LEN = 1
01 - Absolute-length of data : LEN = 1
02 - Data (presumably text in this case) : LEN = 10
Let's pretend that you distinguish chat-packets with the byte 0x63 (99 numerically, and "c" for chat if interpreted as text). Now let's also go with the fact that you said the packet is 12 bytes long, that means the byte at position 01 in our packet will be readable as 10 (12 - (01+1)) by the server and connected clients, so that leaves us with 0x0a (which is numerical byte 10 in hexadecimal).
So far your packet will be:
63 0a __ __ __ __ __ __ __ __ __ __
Now you just replace the empty spaces with some text, I chose "ThisIsEasy"
Now we have our packet of:63 0a 54 68 69 73 49 73 45 61 73 79 <----- Actual packet
-------------------------------------------------
00 01 02 03 04 05 06 07 08 09 10 11 <----- Byte-position
By looking at the byte-position array just above; you can see that it's really just counting out a packet with a size of 12 bytes, yeah?
2) This packet is then sent to the server...
3) The server reads the first two bytes from the packet, which should be 99 (or "c" for chat) and 10 (the length of the data following the header, i.e. directly after the second byte).
4) (OPTIONAL) Since the packet is of the "c"-type (or 99); the server checks the data for inappropriate words, or perhaps admin-commands such as "/ban," etc[sup][1][/sup].
5) Since the packet is of the "c"-type (still 99), we know that it is a local area chat-packet; this means that the server then (at least, it should!) determines the area which the player is in, and sends the (optionally censored) packet out to all other players in the vicinity.
6) No additional processing is needed for this chat-packet, i.e; you're finished, and it's time to move onto the next packet in the queue[sup][2][/sup].
So according to your scenario; if twenty people sent these 12-byte length packets to the server, you will be receiving only 240 bytes. Your server will then know to dispense these chat-packets to the people in the area, and since there are 500: your server will only need to send out 240 * 500 bytes of information. That is 120,000 bytes, and only 117.1875 KB that must be sent out, so you really just multiplied by 500 an extra, unnecessary time
In addition, if you are extra perceptive you will notice that although there are 500 people in the area (which is absurd!), twenty of them are sending messages, and since they are creating the message they already know about it; which means that the server should not send their packet back to them, see it now? So the server only has to send out 240 * 480 bytes of data, or 112.5 KB, this isn't really to reduce the amount of data sent, but rather reduce the amount of processing-time by a very small fraction, as well as prevent them from receiving what they just said (which is pointless and annoying).
Side-notes:
1. If your game implements such administrator-commands; I would hope that you check if the player possesses necessary privileges before actually executing them.
2. The computer which you are hosting such a game on should be powerful enough to almost entirely (if not entirely) avoid having to place packets in a queue. Of course, if the server is programmed inefficiently (or for weaker systems); the way it handles things will be different. The way in which you implement your networking will also result in performance gain/loss.
The side-notes below are not directly attached to text above, so don't try to find them (they are, however; extremely important).
3. The chat-packet used in the example above does not contain the player name, as I reasoned the server should index players by their IPs and accordingly determine the name of the sender from that. You may wish to include the player-name, or other data in the packet, for simplicity, bloat, etc.
4. The scenario which you proposed does not seem very likely, as on very popular side-scroller MMORPGs (MapleStory, Dungeon-fighter Online, etc.) five-hundred players in one area is not a likely occurrence, and to have twenty of them sending messages within one second is also unlikely. This is mainly thanks to the implementation of multiple playable servers and channels (which further separate the servers, but can be played through entirely on one character, whereas that character cannot travel to a separate server).
5. Another option to reduce packet-size is to compress the data before sending it to the server, then decompress and process it there, when (if) sending the final packet out to other players; compress it before doing so. By compressing the string below:
"Hi, I like spamming in online games ;D LALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALALA ;D Happy!"
You could easily end up splitting the size of the needed packet in half, of course, with compression comes the need for a lot more processing (on both the user, and server-end), so you might only want to compress certain types of packets, or just avoid compressing them altogether.
In my opinion; I do not think packet-compression should be used, as transmitting a mere 112 KB per-second is no big deal, even with a 10 Megabit/sec (1.25 Megabyte/sec) upload speed. Popular and high-demand MMORPGs obviously go higher than that, but easily make their money back (plus more) with massive quantities of users.
I do believe that if you want to avoid hackers; it would be best to do a simple XOR over your packets (in case you didn't know; XOR = fast-as-hell), with a certain pattern determined at log-in/whenever.
Author
Sure, id love to use UDP but all the data that is send/received NEEDS to be sent/received and not lost in the process. I don't understand the point of adding a reliability layer to UDP? May as well go for the optimized TCP solution instead like some other MMO's have done. Also I times by 500 again because I am retransmitting data received from all clients back to all clients (so 1 data received is sent to all other 500 clients in this example). If 500 clients are sending data to the server then 500 * 500 is going to be retransmitted back to all connected clients, does that make sence? if not maybe I have calculated wrong.
Spr33 could you please explain to me what XOR is? I heard preventing hackers is basically impossible (and thinking about it, I wouldn't argue against it either, anything is possible and quickly I understood this in a previous thread I made), the best you can do is by making the server check which messages are legit and which are not? That is why I am making the same server-based. Look at games such as LoL - no hackers at ALL and when there is something looking like a hack its just a exploit which can be hot-fixed.
hplus you are right they wont be paying monthly, and I wouldn't even want that sort of system in the first place.
Thinking deeply about this now, I was basically aiming for an indie 2D side-scroller MMO, I somewhat feel like I am completely overestimating myself but at the same time feel like this is completely viable as well, and the prototype I have developed is a example of that.
I seriously doubt 500 players will be in the same area at the same time. Not to mention that sending 20 messages per second seems like a overkill in my calculation as only data is sent when actions occur, I was just stating what the absolute maximum may be and if anything most connected clients in that area would be AFK doing nothing at all. Another thing is that not every type of message will be relayed back to all other clients (i.e. trading with another player does not need to be known by everyone).
Thank you for the replies and continued discussion, it really helps
Any further ideas/thoughts are appreciated,
Xanather.
Spr33 could you please explain to me what XOR is? I heard preventing hackers is basically impossible (and thinking about it, I wouldn't argue against it either, anything is possible and quickly I understood this in a previous thread I made), the best you can do is by making the server check which messages are legit and which are not? That is why I am making the same server-based. Look at games such as LoL - no hackers at ALL and when there is something looking like a hack its just a exploit which can be hot-fixed.
hplus you are right they wont be paying monthly, and I wouldn't even want that sort of system in the first place.
Thinking deeply about this now, I was basically aiming for an indie 2D side-scroller MMO, I somewhat feel like I am completely overestimating myself but at the same time feel like this is completely viable as well, and the prototype I have developed is a example of that.
I seriously doubt 500 players will be in the same area at the same time. Not to mention that sending 20 messages per second seems like a overkill in my calculation as only data is sent when actions occur, I was just stating what the absolute maximum may be and if anything most connected clients in that area would be AFK doing nothing at all. Another thing is that not every type of message will be relayed back to all other clients (i.e. trading with another player does not need to be known by everyone).
Thank you for the replies and continued discussion, it really helps
Any further ideas/thoughts are appreciated,
Xanather.
I highly suggest you rework your packet-handling code to utilize UDP, as it is much, much, much. much, much. much, much faster (and should be used if programming anything that is meant to receive and send packets quickly).
I disagree. There is no *speed* difference between TCP and UDP. The only difference is that, with TCP, if packets are dropped, you have to wait for a re-send before other, received, packets can be delivered, because TCP guarantees in-order delivery.
UDP should be used when low receive latency is more important than ordering, which is the case for internet telephony, and *some* kinds of games.
Ehh... This is generally frowned upon
4) Data is only sent when someone actually DOES something
[/quote]
Generally frowned upon by whom, and why? Not sending data you don't have to send is generally approved by all network programmers I know.
Why do you multiply by 500 twice?
[/quote]
Because, as he said, he's broadcasting messages from 500 players, to 500 players. This is the basic n-squared growth of traffic that's the core of all MMO systems.
I do believe that if you want to avoid hackers; it would be best to do a simple XOR over your packets (in case you didn't know; XOR = fast-as-hell), with a certain pattern determined at log-in/whenever.
[/quote]
First, the speed of XOR scrambling versus a real cypher doesn't matter much. The available bandwidth on a computer limits how many bytes need to be encrypted per second, which in turn means that the overhead of the encryption is very small compared to available CPU power.
Second, scrambling with XOR instead of using a real cypher is a really bad idea. If your data needs encryption, you should encrypt it, not just cover it in the thinnest of layers of paint.
Third, you cannot really protect against hackers by encrypting your data anyway, because the hacker has full control of the client machine.
I would try to keep the amount of data you send out to less than 5kiB/s per connection. Many badly designed multiplayer games achieve that, well-designed games often run with 1/2 as much. 117kiB/s is insane, if for no other reason then because at a typical "average" home user internet speed of 2-3Mib/s it takes upwards of half a second to transmit this bulk (and, as hplus0603 pointed out, bandwidth is not free -- you do get a gigabit link for as little as 69€ additional setup in some places, but if you send out terabytes of data every month you will still need pay for that). Half a second of lag added "for nothing" is a lot.
About XOR-encryption as suggested by Spr33, this is OK for making the data stream look random to the casual eye, but no more. It doesn't cost significantly more to add a "real" encryption, any readily available well-known and tested implementation will do (which, again, probably doesn't perform much better because most likely you'll do the key exchange in an insecure manner (few people do this right), but still it's silly to try and economize on the encryption).
UDP is no viable alternative to TCP in "making stuff faster" here. UDP is "faster" because it sends independent datagrams without something like Nagle's algorithm and does not guarantee reliable in-order delivery. You've already disabled Nagle, so 90% of TCP's slowness is already gone. The problem here is that 117kiB have to go down the wire, and UDP still has to do that too. The actual speed is exactly the same.
About XOR-encryption as suggested by Spr33, this is OK for making the data stream look random to the casual eye, but no more. It doesn't cost significantly more to add a "real" encryption, any readily available well-known and tested implementation will do (which, again, probably doesn't perform much better because most likely you'll do the key exchange in an insecure manner (few people do this right), but still it's silly to try and economize on the encryption).
UDP is no viable alternative to TCP in "making stuff faster" here. UDP is "faster" because it sends independent datagrams without something like Nagle's algorithm and does not guarantee reliable in-order delivery. You've already disabled Nagle, so 90% of TCP's slowness is already gone. The problem here is that 117kiB have to go down the wire, and UDP still has to do that too. The actual speed is exactly the same.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement