69 replies to this topic

[canned]

I'm working on an MMORPG style game for a school project. It is a client-server model with a database as a backend. I'm wondering what protocols should be used for what. I know the UDP vs TCP argument is a common question and is discussed in the FAQ. I need to take it a little further than what is covered there. The game is ship based and will involve combat between ships. So, for ship movement and cannon ball movement, ie the twitch gaming portions of the game, I'm thinking UDP is the best way to go (as seems to be the common approach). So I'll use UDP and interpolation to handle that. As for the other elements of the game, I'm wondering if TCP is the best way to go, thus using both protocols. First of all, is it common that games will use both protocols, or do they often times pick one and settle on it. Is there a performance motivation to settling on one packet or using both. The other elements of the game: Users will dock at ports. The user then requests the port information from the server and displays it when its received. Port information is sent only on a request-basis and thus, a lost packet cannot merely be ignored in favor or a later update. Users will fire cannon balls in combat and will need to tell the server that such an event has occurred. Once cannon balls are fired, the cannons must be reloaded. Thus, if the user fired the cannons and the packet declaring this never made it to the server, the user's cannons would be unloaded and they would have to wait for them to reload again without ever reaping the benefits of the lost shot. There will be other similar events in the game. Is using both TCP and UDP for this game a good model to follow? Or would people suggest one over the other. Next, I have a few questions that are specific to the MMORPG genre. I'm not sure how to handle things like user inventory and statistics. Do you maintain a copy of the users inventory on the client side, only updating the server when the inventory changes? Or do you store the inventory on the server and fetch it any time the user requests to see their inventory? The quick answer that comes to mind for me is that it would be best to manage it on the user side and just update the server as necessary. This seems to achieve the least amount of server interaction and ensures that things happen in a timely manner on the client side. However, it seems more prone to hacking (Like the Diablo complex)... So how do games handle issues like that? I know validating any information that is given to the server is important, but are there any other tricks?

[Scott Markwell]

TCP would be my recomendation because you can't pull the Quake3 trick of sending a time encoded state package, because if it is a MMO, then there is alot of data that NEEDS to get to it's destination (I want to do x/y/z, and don't want to deal with race conditions, or having to resend data).

Anyone have the look at Quake3 network code story hanging around? It's an intresting read, but it only really works for small enviroments where the priority is speed and not efficiancy or realiablity. Mind you, that system can be used over TCP or UDP. Just TCP automatically made sure the data arrived in a certain order.

EDIT: Found it
http://www.bookofhook.com/Article/GameDevelopment/TheQuake3NetworkingModel.html

Also See the Forum FAQ about TCP v UDP
http://www.gamedev.net/community/forums/showfaq.asp?forum_id=15

So anyways: TCP

[canned]

Oops! Forgot to log in for the last post... So just FYI, the last post from Anonymous was from me (the original author of the question).

[Scott Markwell]

Doing two may be benificial, but I'd suggest working with TCP now, leaving in hooks for UDP later. I think it would be more benificial to get the project up and running then worrying about latancy issues this early. As pointed out in that Quake 3 article I linked, doing both TCP and UDP at the same time introduces intresting and very hard to track bugs.

[wazoo69]

I've been hearing that World of Warcraft is done with TCP for their communication.
(It seems fast enough to me *wink*)

Probably also helps to deal with security/hacking issues that are much harder to track/prevent with straight UDP..

[hplus0603]

Quote:

(It seems fast enough to me *wink*)

Yeah, that's why the FAQ talks about the "speed" of the two protocols.

Rumor has it that one of the reasons they had such lag and server queue problems initially was that they were using TCP, and the number of connections per server was just too much. With TCP, there is overhead per connected user in the form of a separate socket; with UDP, you can push that into user space and a simple hash table.

For the project described here, however, I doubt that either protocol will get in the way; pick one (or better: pick an existing library) and run with it.

[Megahertz]

They still are using TCP afaik. Unless something has changed in the last two weeks. I'll check again when I get home.

Also the lag issues are still present. Whether this is a result of using TCP or just the high volume of users, whos to say other than Blizzard.

I have ran some TCPdump's on the game while playing and one thing that I noticed is when you use the mouse to "look around" the client pelts the server with an insane number of updates/sec. I didnt count, but I'd say on the order of 30 packets/sec.

With that said, TCP seems to be working good for the most part.


-=[ Megahertz ]=-

[The Forgotten Mindset]

RakNet uses UDP, but I gives you all the advantages of TCP through software.

It allows you to send priority messages, packet ordering, etc.

rekkarsoft.com
GameDev Product Review

I'd really suggest using it. I'm doing the same thing for my school project (actually 2 school projects), except my game is tile-based.

[joew]

Depends on the problem space and type of game.

RunUO is an Ultima Online emulator written in C# that has had up to 3200 connected clients on a quad Xeon box. Of course an isometric point and click game uses a hell of a lot less traffic (movement/view) than a 3D game.

You could also look at Dawn of Light which is a Dark Age of Camelot emulator written in C# that is a TCP/UDP hybrid approach. Basically the movement system works off UDP and for things that need to get to the server TCP works well.

So UO uses TCP, WoW uses TCP, Camelot uses TCP w/ UDP for movement, AC uses UDP, City of Heroes UDP, EQ TCP w/ UDP.

I'm not so sure if it is fair to say that WoW is laggy because of using TCP over UDP quite yet unless we see them change network subsystems.. as AC had a hell of a lot more lag than WoW ever has but it is using UDP.

So really it is up to the problem space, at least in my opinion I may be wrong on this.

[graveyard filla]

hey everyone,

Saruman's post kind of brought up some confusion... from what I've heard from the experts around here, UDP should always be used with MMORPG's simply because of the scalability problems associated with TCP... so, whats the deal? [grin].

[joew]

Quote:

Original post by graveyard filla hey everyone, Saruman's post kind of brought up some confusion... from what I've heard from the experts around here, UDP should always be used with MMORPG's simply because of the scalability problems associated with TCP... so, whats the deal? [grin].

Sorry I don't mean to be confusing, I was just stating what commercial games use and for the most part MMOG games have been based off TCP in the past.

I have a larger listing at home that I will try to dig up, but the games and networking subsystems I listed above are correct. I actually have a server for a prototype game that can easily handle 2000-3000 clients on a single machine, but obviously it is not scalable and once I add in more defined movement/views I will need more than a single server system. I was looking at a variety of solutions but it is a very hard problem set to address as well as implement. I still haven't decided on the way we are moving with our product, but it has come down to either licensing a middleware product or using our current solution but zone-based.

Scalability completely depends on your architecture.. obviously if you have one single main server that hosts 3000+ TCP connections you will bog down... but if you have 5 of that machine talking with other ones, I haven't seen an issue yet. So far all my testing and prototype tweaking has lead me to do is use 2 servers in front and one single simulation server, which seems very nice for scalability.. except scaling the simulation sever needs to be thought out and is an amazingly hard problem set.

Oh Anarchy Online is also using a TCP based network architecture. Actually the majority that I have seen or have information about use a TCP based network architecture, or a TCP based architecture with UDP for quick updates such as movement. But as I stated it really depends on the problem set, as City of Heroes could not have been created using the same mechanics using TCP.

[Edited by - Saruman on May 12, 2005 8:40:36 PM]

[graveyard filla]

Thanks for the reply... I understand what you're saying, and it sounds interesting... but if UDP offers better scalability then why even bother? Is there any real advantage other then the fact that the low level protocol stuff is handled for you (e.g. reliability)?

I know what you mean about how complex a distributed simulation could be... from my own designs and talking with people like hplus, it seems like a very huge beast to tackle... so many problems with hand offs and such... my best idea was to just make it zone based, which is probably the simplest method... although this can only scale so well..

What's really interesting to me is a seamless, truly scalable world which isnt based on zones... e.g. two people standing next to each other in game can actually be connected to 2 seperate servers, for true scalability (imaging thousands of people all crowded in a small area and killing each other, this would be possible given the hardware with such an architecture)... IIRC 'There' works like this... some day hplus is gunna have to write a paper on all this crap [grin].

[joew]

I may be completely wrong on this and hplus might have to correct it, but I have not yet seen enough proof to say that UDP is more scalable as a layer to build your architecture upon. If it was than why would the majority of these games use TCP?

If you are going to spam packets such as movement.. than UDP seem the obvious choice to me because there is less overhead. But for anything else I fail to see why TCP is not able to scale or be used.

I also agree with you on how awesome the technology used in 'There' is, it is quite amazing. Although I would be willing to put money on the fact that is not a technology that will become 'mainstream' or 'public knowledge' really as it takes a lot more than a single person to build load balanced simulation servers.

One design we have been tossing around and are thinking of prototyping is basically our TCP simulation server but breaking apart certain components to be distributed services. We want to split apart the movement to a UDP service which would allow us to scale movement very easily even though it is all interacting with a single simulation server. Sure this doesn't give us ultimate scalability, but if we do that for each 'zone' it makes zones able to handle a ton of traffic. I mean 60-70% of traffic is movement traffic, so if we can offload that to a seperate service and have the other 30% on the simulation server I would be very happy.

[joew]

I should have stated the other alternative that we were looking at and is still possible. We were looking at using RakNet as a viable networking solution for our project although there are some major issues with handling a large number of players at the current time. We have talked to Kevin Jenkins about this and know exactly where the problems are and will be looking into this if we do select it as our approach. Not to say RakNet is a bad library, I actually find it very good.. it just needs some work for having a large number of players online.

Currently this is something we are looking at to solve our approach of having to use both TCP and UDP which can cause some trouble.. but we will either be using that approach or a full RakNet approach although we will have to modify the library to get it working for a large number of players.

[hplus0603]

If you need real-time response to movement, then UDP is likely better. If your MMO is a simulation (similar to an FPS), then you need real-time response. AFAIK, There and City of Heroes are both simulations (more like an FPS).

If your MMO is more of a classic MUD/RPG, and not a full "distributed simulation" then you don't necessarily need real-time properties in your protocol. You might be OK with receiving older data as part of a re-transmit. In that case, TCP will work fine, as long as the amount of connections doesn't overwhelm your server machine. With current hardware, I'd believe you can do 3,000 clients on a single machine, for RPG/MUD-style interactions.

So, analyze your problem; pick the right architecture.

Quote:

some day hplus is gunna have to write a paper on all this crap

I did. Sadly, we currently require an NDA to send it to you. Apparently, defense contracting isn't quite as open an environment as open source game development yet.

[JJacobo]

We are actually tackling this same problem with our MMORPG project. Since our engine is more action/twich-like in terms of combat, movement, speech (a headache in its own regard!)we will be using UDP for most of the system but we are looking into applying TCP for other tasks such as item interaction, property interaction, and player statistics (though the system is very much opaque). Several of our coders have come from MMORPG EMU backgrounds where the combination was oftentimes a necessity.

HP, is right about UDP in that it takes much more processing power so one of the things we're working on is implementing dynamic zoning, so that the players are seamlessly interacting with one another without overloading the servers by having everyone's slightest movement detail broadcasted to the whole map!

I am not a coder myself but if you have any questions about how we're planning on doing it email or PM me. 0_o

[hplus0603]

Quote:

we will be using UDP for most of the system but we are looking into applying TCP for other tasks such as item interaction

If you've already built infrastructure for UDP communication, I recommend sending everything over it. We use UDP/TCP, and it works, but we had to do a LOT of mitigation to avoid large TCP packets from introducing unnecessary jitter on the UDP connection, especially for modem users.

I'd rather have a single point of communication, which can totally manage the bandwidth to the client, for all the data streams. Each data stream managing itself (as in a single TCP stream) is not ideal.

[John Schultz]

Quote:

Original post by hplus0603 Quote: we will be using UDP for most of the system but we are looking into applying TCP for other tasks such as item interaction If you've already built infrastructure for UDP communication, I recommend sending everything over it. We use UDP/TCP, and it works, but we had to do a LOT of mitigation to avoid large TCP packets from introducing unnecessary jitter on the UDP connection, especially for modem users. I'd rather have a single point of communication, which can totally manage the bandwidth to the client, for all the data streams. Each data stream managing itself (as in a single TCP stream) is not ideal.

My early networked systems used UDP and TCP (including linked COTS hardware, multi-head (view) sims, with no genlocking (worked great)). Really early networked code (before the internet (arpanet was around)) it was all modems, and custom CRC-based lock-step protocols were the best you could do.

I started out using UDP+TCP for an Xbox launch title, but due to early performance issues, was forced to create a custom protocol using UDP only. The custom protocol effectively combines UDP and TCP into UDP-only packets. This is the most efficient method possible, especially for low bandwidth apps. I tested 100 moving 3D objects on a laptop connected via cellphone (14.4kbps data rate), and it worked fine (ended up around 1000ms latency).

Thus I recommend getting everything working with UDP+TCP until you hit a wall and are forced to write a custom UDP protocol. At that point you'll also have a much better idea of how to tackle the creation of a custom reliable UDP-based protocol (at least a few weeks to develop and possibly months to debug and test). You could also switch to a low cost or free existing library (Raknet, etc.).

It's not clear if this is still true, but in 2002, some internet routers gave preference to UDP packets over TCP packets. That is, during high traffic situations, TCP packets would be dropped in preference to UDP packets, with the logic being that UDP packets are time critical for real-time apps, whereas TCP was carrying mostly non-time-sensitive data. I poked around google for info in 2005, but didn't find anything pointing one way or the other as to how companies are actually configuring their internet routers (a place to look would be Cisco tech info, etc.).

IP Packet Dropping Policy.
See the links at bottom of this page.
Packet dropping techniques for multimedia, RED (Random Early Detection) and Weighted RED (WRED).
Cisco's WRED.

For a large MMOG, unless the custom UDP protocol has excellent bandwidth adaption, TCP may be a better choice for keeping data flowing smoothly. This is one reason why custom UDP protocol authors should study TCP very carefully during the design phase (see my previous posts for links to excellent TCP/design/implementation papers).

[Alamar]

I didn't read all the posts, so this may have been said already.

TCP vs UDP isn't the only issue/concern with performance/scalability. There are also issues of synchronisity (that a word?) and overlapping.

I won't make a grand point here, but if you haven't already, take a look at IO Completion ports. They're a winsock 2 method of handling very many (thousands) connections more efficiently (and requires an NT based system). They're inherently multithreaded though, so consider that when you decide what to do : )

-Alamar