hey everyone,
Saruman's post kind of brought up some confusion... from what I've heard from the experts around here, UDP should always be used with MMORPG's simply because of the scalability problems associated with TCP... so, whats the deal? [grin].
MMORPG and the ol' UDP vs TCP
Quote:Original post by graveyard filla
hey everyone,
Saruman's post kind of brought up some confusion... from what I've heard from the experts around here, UDP should always be used with MMORPG's simply because of the scalability problems associated with TCP... so, whats the deal? [grin].
Sorry I don't mean to be confusing, I was just stating what commercial games use and for the most part MMOG games have been based off TCP in the past.
I have a larger listing at home that I will try to dig up, but the games and networking subsystems I listed above are correct. I actually have a server for a prototype game that can easily handle 2000-3000 clients on a single machine, but obviously it is not scalable and once I add in more defined movement/views I will need more than a single server system. I was looking at a variety of solutions but it is a very hard problem set to address as well as implement. I still haven't decided on the way we are moving with our product, but it has come down to either licensing a middleware product or using our current solution but zone-based.
Scalability completely depends on your architecture.. obviously if you have one single main server that hosts 3000+ TCP connections you will bog down... but if you have 5 of that machine talking with other ones, I haven't seen an issue yet. So far all my testing and prototype tweaking has lead me to do is use 2 servers in front and one single simulation server, which seems very nice for scalability.. except scaling the simulation sever needs to be thought out and is an amazingly hard problem set.
Oh Anarchy Online is also using a TCP based network architecture. Actually the majority that I have seen or have information about use a TCP based network architecture, or a TCP based architecture with UDP for quick updates such as movement. But as I stated it really depends on the problem set, as City of Heroes could not have been created using the same mechanics using TCP.
[Edited by - Saruman on May 12, 2005 8:40:36 PM]
Thanks for the reply... I understand what you're saying, and it sounds interesting... but if UDP offers better scalability then why even bother? Is there any real advantage other then the fact that the low level protocol stuff is handled for you (e.g. reliability)?
I know what you mean about how complex a distributed simulation could be... from my own designs and talking with people like hplus, it seems like a very huge beast to tackle... so many problems with hand offs and such... my best idea was to just make it zone based, which is probably the simplest method... although this can only scale so well..
What's really interesting to me is a seamless, truly scalable world which isnt based on zones... e.g. two people standing next to each other in game can actually be connected to 2 seperate servers, for true scalability (imaging thousands of people all crowded in a small area and killing each other, this would be possible given the hardware with such an architecture)... IIRC 'There' works like this... some day hplus is gunna have to write a paper on all this crap [grin].
I know what you mean about how complex a distributed simulation could be... from my own designs and talking with people like hplus, it seems like a very huge beast to tackle... so many problems with hand offs and such... my best idea was to just make it zone based, which is probably the simplest method... although this can only scale so well..
What's really interesting to me is a seamless, truly scalable world which isnt based on zones... e.g. two people standing next to each other in game can actually be connected to 2 seperate servers, for true scalability (imaging thousands of people all crowded in a small area and killing each other, this would be possible given the hardware with such an architecture)... IIRC 'There' works like this... some day hplus is gunna have to write a paper on all this crap [grin].
I may be completely wrong on this and hplus might have to correct it, but I have not yet seen enough proof to say that UDP is more scalable as a layer to build your architecture upon. If it was than why would the majority of these games use TCP?
If you are going to spam packets such as movement.. than UDP seem the obvious choice to me because there is less overhead. But for anything else I fail to see why TCP is not able to scale or be used.
I also agree with you on how awesome the technology used in 'There' is, it is quite amazing. Although I would be willing to put money on the fact that is not a technology that will become 'mainstream' or 'public knowledge' really as it takes a lot more than a single person to build load balanced simulation servers.
One design we have been tossing around and are thinking of prototyping is basically our TCP simulation server but breaking apart certain components to be distributed services. We want to split apart the movement to a UDP service which would allow us to scale movement very easily even though it is all interacting with a single simulation server. Sure this doesn't give us ultimate scalability, but if we do that for each 'zone' it makes zones able to handle a ton of traffic. I mean 60-70% of traffic is movement traffic, so if we can offload that to a seperate service and have the other 30% on the simulation server I would be very happy.
If you are going to spam packets such as movement.. than UDP seem the obvious choice to me because there is less overhead. But for anything else I fail to see why TCP is not able to scale or be used.
I also agree with you on how awesome the technology used in 'There' is, it is quite amazing. Although I would be willing to put money on the fact that is not a technology that will become 'mainstream' or 'public knowledge' really as it takes a lot more than a single person to build load balanced simulation servers.
One design we have been tossing around and are thinking of prototyping is basically our TCP simulation server but breaking apart certain components to be distributed services. We want to split apart the movement to a UDP service which would allow us to scale movement very easily even though it is all interacting with a single simulation server. Sure this doesn't give us ultimate scalability, but if we do that for each 'zone' it makes zones able to handle a ton of traffic. I mean 60-70% of traffic is movement traffic, so if we can offload that to a seperate service and have the other 30% on the simulation server I would be very happy.
I should have stated the other alternative that we were looking at and is still possible. We were looking at using RakNet as a viable networking solution for our project although there are some major issues with handling a large number of players at the current time. We have talked to Kevin Jenkins about this and know exactly where the problems are and will be looking into this if we do select it as our approach. Not to say RakNet is a bad library, I actually find it very good.. it just needs some work for having a large number of players online.
Currently this is something we are looking at to solve our approach of having to use both TCP and UDP which can cause some trouble.. but we will either be using that approach or a full RakNet approach although we will have to modify the library to get it working for a large number of players.
Currently this is something we are looking at to solve our approach of having to use both TCP and UDP which can cause some trouble.. but we will either be using that approach or a full RakNet approach although we will have to modify the library to get it working for a large number of players.
If you need real-time response to movement, then UDP is likely better. If your MMO is a simulation (similar to an FPS), then you need real-time response. AFAIK, There and City of Heroes are both simulations (more like an FPS).
If your MMO is more of a classic MUD/RPG, and not a full "distributed simulation" then you don't necessarily need real-time properties in your protocol. You might be OK with receiving older data as part of a re-transmit. In that case, TCP will work fine, as long as the amount of connections doesn't overwhelm your server machine. With current hardware, I'd believe you can do 3,000 clients on a single machine, for RPG/MUD-style interactions.
So, analyze your problem; pick the right architecture.
I did. Sadly, we currently require an NDA to send it to you. Apparently, defense contracting isn't quite as open an environment as open source game development yet.
If your MMO is more of a classic MUD/RPG, and not a full "distributed simulation" then you don't necessarily need real-time properties in your protocol. You might be OK with receiving older data as part of a re-transmit. In that case, TCP will work fine, as long as the amount of connections doesn't overwhelm your server machine. With current hardware, I'd believe you can do 3,000 clients on a single machine, for RPG/MUD-style interactions.
So, analyze your problem; pick the right architecture.
Quote:some day hplus is gunna have to write a paper on all this crap
I did. Sadly, we currently require an NDA to send it to you. Apparently, defense contracting isn't quite as open an environment as open source game development yet.
We are actually tackling this same problem with our MMORPG project. Since our engine is more action/twich-like in terms of combat, movement, speech (a headache in its own regard!)we will be using UDP for most of the system but we are looking into applying TCP for other tasks such as item interaction, property interaction, and player statistics (though the system is very much opaque). Several of our coders have come from MMORPG EMU backgrounds where the combination was oftentimes a necessity.
HP, is right about UDP in that it takes much more processing power so one of the things we're working on is implementing dynamic zoning, so that the players are seamlessly interacting with one another without overloading the servers by having everyone's slightest movement detail broadcasted to the whole map!
I am not a coder myself but if you have any questions about how we're planning on doing it email or PM me. 0_o
HP, is right about UDP in that it takes much more processing power so one of the things we're working on is implementing dynamic zoning, so that the players are seamlessly interacting with one another without overloading the servers by having everyone's slightest movement detail broadcasted to the whole map!
I am not a coder myself but if you have any questions about how we're planning on doing it email or PM me. 0_o
Quote:we will be using UDP for most of the system but we are looking into applying TCP for other tasks such as item interaction
If you've already built infrastructure for UDP communication, I recommend sending everything over it. We use UDP/TCP, and it works, but we had to do a LOT of mitigation to avoid large TCP packets from introducing unnecessary jitter on the UDP connection, especially for modem users.
I'd rather have a single point of communication, which can totally manage the bandwidth to the client, for all the data streams. Each data stream managing itself (as in a single TCP stream) is not ideal.
Quote:Original post by hplus0603Quote:we will be using UDP for most of the system but we are looking into applying TCP for other tasks such as item interaction
If you've already built infrastructure for UDP communication, I recommend sending everything over it. We use UDP/TCP, and it works, but we had to do a LOT of mitigation to avoid large TCP packets from introducing unnecessary jitter on the UDP connection, especially for modem users.
I'd rather have a single point of communication, which can totally manage the bandwidth to the client, for all the data streams. Each data stream managing itself (as in a single TCP stream) is not ideal.
My early networked systems used UDP and TCP (including linked COTS hardware, multi-head (view) sims, with no genlocking (worked great)). Really early networked code (before the internet (arpanet was around)) it was all modems, and custom CRC-based lock-step protocols were the best you could do.
I started out using UDP+TCP for an Xbox launch title, but due to early performance issues, was forced to create a custom protocol using UDP only. The custom protocol effectively combines UDP and TCP into UDP-only packets. This is the most efficient method possible, especially for low bandwidth apps. I tested 100 moving 3D objects on a laptop connected via cellphone (14.4kbps data rate), and it worked fine (ended up around 1000ms latency).
Thus I recommend getting everything working with UDP+TCP until you hit a wall and are forced to write a custom UDP protocol. At that point you'll also have a much better idea of how to tackle the creation of a custom reliable UDP-based protocol (at least a few weeks to develop and possibly months to debug and test). You could also switch to a low cost or free existing library (Raknet, etc.).
It's not clear if this is still true, but in 2002, some internet routers gave preference to UDP packets over TCP packets. That is, during high traffic situations, TCP packets would be dropped in preference to UDP packets, with the logic being that UDP packets are time critical for real-time apps, whereas TCP was carrying mostly non-time-sensitive data. I poked around google for info in 2005, but didn't find anything pointing one way or the other as to how companies are actually configuring their internet routers (a place to look would be Cisco tech info, etc.).
IP Packet Dropping Policy.
See the links at bottom of this page.
Packet dropping techniques for multimedia, RED (Random Early Detection) and Weighted RED (WRED).
Cisco's WRED.
For a large MMOG, unless the custom UDP protocol has excellent bandwidth adaption, TCP may be a better choice for keeping data flowing smoothly. This is one reason why custom UDP protocol authors should study TCP very carefully during the design phase (see my previous posts for links to excellent TCP/design/implementation papers).
I didn't read all the posts, so this may have been said already.
TCP vs UDP isn't the only issue/concern with performance/scalability. There are also issues of synchronisity (that a word?) and overlapping.
I won't make a grand point here, but if you haven't already, take a look at IO Completion ports. They're a winsock 2 method of handling very many (thousands) connections more efficiently (and requires an NT based system). They're inherently multithreaded though, so consider that when you decide what to do : )
-Alamar
TCP vs UDP isn't the only issue/concern with performance/scalability. There are also issues of synchronisity (that a word?) and overlapping.
I won't make a grand point here, but if you haven't already, take a look at IO Completion ports. They're a winsock 2 method of handling very many (thousands) connections more efficiently (and requires an NT based system). They're inherently multithreaded though, so consider that when you decide what to do : )
-Alamar
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
