• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
canned

MMORPG and the ol' UDP vs TCP

69 posts in this topic

Quote:
Original post by Anonymous Poster
TCP is a reliable protocol. Although this sounds like I'm paying it a compliment, it can be a bad thing for games.

Reliable means that the packets you send are received in the same order. If you send packets:

X -> Y -> Z they are received as X -> Y -> Z

This is great for chat/communication, but horrible for just about anything else.
In UDP packets are received in the order that they arrive. Which is great for movement, or actions, because the last action is less important than the current action. You can mitigate the problems with out of order sequences on UDP very easily, simply by queuing the messages as they arrive, and then re-requesting those that didn't make it. Once your packets are ordered, then you can process them.

Unfortunately there is no mitigation for the reliability of TCP. If a packet is dropped, then no new packets will be accepted until the missing packet arrives. This can cause a great amount of perceived lag. Have you ever played a game where you are moving your sprite, and then all of a sudden you stop moving, so you keep hitting the forward button. Then, all of the sudden your sprite bursts into a fast-forward sprint. This is because the movement packets were sent with TCP.

I would strongly suggest that you use a network API that mitigates these problems for you. Most of them do. I see no feasible reason to use TCP, except to be lazy.



Good luck.



The above issues apply to all reliable protocols, including those built into custom reliable-UDP protocols. Such issues are network game design issues, as opposed to network protocol issues. The moving sprite example problem can be solved with local client simulation and server reconciliation.

UDP+TCP (with throttling) is a reasonable choice for a first time network programmer working on a game project. A custom reliable-UDP protocol can only be more efficient (it can't be more reliable (a boolean trait)). Packet ordering and reliable system design are relatively straightforward. Creating a bandwidth efficient system in addition to the reliable system is not trivial.

TCP is an excellent protocol. The advantage to a custom UDP system is the ability to optimize performance and to have total control of system behavior. TCP could have been more efficient for game/simulation applications if more TCP network options were available and/or more options were standardized across platforms.

Again, a good reliable-UDP protocol will emulate TCP very closely, only differing where more control and efficiency can be gained for game/simulation applications.

UDP's biggest strength for a shipping product may be the firewall issue (worst case, only 1 (or a few) fixed ports must be opened). This is less of an issue for a MMOG with a centralized server (never hosted by other players), as fixed TCP ports can be used (including port 80 (HTTP): it will always work (unless HTTP is specifically blocked)).
0

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous Poster
In UDP packets are received in the order that they arrive.


This isn't true. UDP can send packets out of order, and in fact can send duplicates and other nasty things as well.

BTW, this thread is really making me want to fire up my IDE and pick up where i left off with FTA... if I could even remember where that was [grin]. I was just in the middle of switching movement over to use the keyboard... unfortunately real life is taking over lately and I don't have time for that stuff right now... deadline this tuesday [grin].
0

Share this post


Link to post
Share on other sites
Quote:
Original post by Saruman
I should have stated the other alternative that we were looking at and is still possible. We were looking at using RakNet as a viable networking solution for our project although there are some major issues with handling a large number of players at the current time. We have talked to Kevin Jenkins about this and know exactly where the problems are and will be looking into this if we do select it as our approach. Not to say RakNet is a bad library, I actually find it very good.. it just needs some work for having a large number of players online.

Currently this is something we are looking at to solve our approach of having to use both TCP and UDP which can cause some trouble.. but we will either be using that approach or a full RakNet approach although we will have to modify the library to get it working for a large number of players.


Do you think you could go more into details about this? Ive looked through a lot of the RakNet source, and it seems very elegant... although why he made all his own data structures is beyond me... What's wrong with RakNet right now that doesn't allow large amounts of players? And could you define 'large'?
0

Share this post


Link to post
Share on other sites
Quote:
The above issues apply to all reliable protocols, including those built into custom reliable-UDP protocols.


Not so. If you build a state-reliable protocol (rather than a stream-reliable protocol), then you can deliver things out of order just fine, because the promise of a state reliable protocol is that the mirrored state will eventually be consistent with the sent state. For a stream-reliable protocol (such as TCP), the promise is that the *data* sent is the same on the receiving end as the sending end, which is a much stronger guarantee (and unnecessarily strong for networked games).

Also, there are classes of reliability such as "integrity and sequencing but not lossless" which are quite useful to games. That semantic is typically available in a "reliable" UDP wrapper, but can't be provided by TCP.
0

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
Quote:
The above issues apply to all reliable protocols, including those built into custom reliable-UDP protocols.


Not so. If you build a state-reliable protocol (rather than a stream-reliable protocol), then you can deliver things out of order just fine, because the promise of a state reliable protocol is that the mirrored state will eventually be consistent with the sent state. For a stream-reliable protocol (such as TCP), the promise is that the *data* sent is the same on the receiving end as the sending end, which is a much stronger guarantee (and unnecessarily strong for networked games).

Also, there are classes of reliability such as "integrity and sequencing but not lossless" which are quite useful to games. That semantic is typically available in a "reliable" UDP wrapper, but can't be provided by TCP.


The above quoted statement, taken in its original context, referred to the fact that the issue was a game design issue, and not a reliable protocol issue. In the case of the example problem under discussion, player movement keystrokes would have to be processed in order, otherwise the state would rapidly diverge.

On the topic of out-of-order reliable support (OOORS): while I agree with some of your points regarding levels of reliability, in practice I haven't seen any evidence that a custom reliable UDP protocol that supports OOOR packets has a signficant advantage over TCP, provided everything else is equal (both have similar behavior to bandwidth adaption, etc.). While all packets should be CRC'd (integrity), and sequencing support (ordered data) is indeed useful for movement data, I would not classify these concepts as reliable by themselves. CRC and sequence support can be applied to straight UDP (e.g. movement data) alongside TCP for reliable messages (critical state only).

I can see the benefit of having a power-up/game-item-activation happening with immediate feedback being considered a benefit of OOORS, but given the infrequent occurance of OOO packets, I don't see OOORS being a strong argument (by itself) for a custom reliable UDP protocol over TCP. One argument for OOORS over TCP would be a reliable signaling packet, so that slow clients can be instructed to perform special actions (such as reset to a known state so that the server can reset the client back to a (last) known synchronized state, allowing the server to flush/reset it's send queue(s)). While TCP provides for out-of-band data support (OOB), it is not supported on all TCP implementations. However, a second TCP critical message channel (that takes near zero bandwidth) can provide equivalent behavior.

In my network simulations and internet tests, I did not see a major performance issue with out of order packets. Additionally, given that certain (major) encrypted and authenticated protocols drop all out of order packets, there was no point in implementing a protocol that has to worry about out of order packets (in one particular project, a lower level network layer dropped all out of order packets).

While I'm defending TCP, I also encourage developers to explore creating custom reliable UDP protocols, tailored to their needs. However, if the custom UDP protocol provides no real benefit over TCP in practice (or performs worse than TCP, on average (from low- to high- congestion situations, i.e. does not scale well), there's no point in reinventing the wheel, especially if the new wheel ends up being square.
0

Share this post


Link to post
Share on other sites
The problem is that, if I send the state for object A over TCP at time T, and there's packet loss and re-transmission, the TCP stack will send another copy of A at T across the wire, even though A has since evolved to time T+dt. With a custom protocol, you can at that point send the state of A at T+dt, so that re-transmits are more efficient.

Similarly, for a single object it may make sense to deliver messages in order, but trying to enforce that across objects will lead to the entire world hitching when a single packet is dropped (not just the objects who had updates in the packet). You may think that a separate TCP connection per object would avoid this, but then instead you have the full packet overhead of TCP per packet per object, which is a rather high price to pay -- you can do much better yourself over UDP.

From my personal experience, these are two of the three major reasons to use UDP. The third reason is that UDP can do peer-to-peer punch-through through NAT. This recommendation does, however, assume that you know what you're doing... not only with congestion avoidance, but with things like security and authentication as well!
0

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
The problem is that, if I send the state for object A over TCP at time T, and there's packet loss and re-transmission, the TCP stack will send another copy of A at T across the wire, even though A has since evolved to time T+dt. With a custom protocol, you can at that point send the state of A at T+dt, so that re-transmits are more efficient.


Right: send such data using UDP in the unreliable channel. Stale data is thus never retransmitted. Only transmit data that absolutely must arrive via the TCP channel.

Quote:
Original post by hplus0603
Similarly, for a single object it may make sense to deliver messages in order, but trying to enforce that across objects will lead to the entire world hitching when a single packet is dropped (not just the objects who had updates in the packet). You may think that a separate TCP connection per object would avoid this, but then instead you have the full packet overhead of TCP per packet per object, which is a rather high price to pay -- you can do much better yourself over UDP.


That is a good argument for OOORS for UDP when the simulation requires objects to be updated in such a manner. However, the game state management can be designed around the player's view space, for example, and there really shouldn't be a big difference in object update order (in terms of what the player will experience). The point is that there should never be a huge backlog of queued data that a "hold up / ordering" case causes a diminished player experience. This can be mathematically modeled to a limit case based on network bandwidth and maximum allowed latency for game/simulation effects. Put another way, if the player is experiencing delayed effects due to a "hold up / ordering" issue in the reliable channel, the data rate is too high.

Quote:
Original post by hplus0603
From my personal experience, these are two of the three major reasons to use UDP. The third reason is that UDP can do peer-to-peer punch-through through NAT. This recommendation does, however, assume that you know what you're doing... not only with congestion avoidance, but with things like security and authentication as well!


I agree with the P2P NAT point, and as stated previously, I use UDP exclusively for my current game/simulation projects. However, I can understand why some MMOG developers chose TCP when the game/simulation resides on a dedicated server. In such a case, all I/O can go through port 80 (HTTP) or even 443 (HTTPS). Thus, there will never be issues with NAT/firewalls (unless HTTP/HTTPS is blocked, as in some corporate environments where web access is not allowed).

On the topic of security and authentication: Kerberos and derivatives are a good choice. Full, free source code is available here.

Per Bruce Schneier, nothing is ever fully secure: always changing. Applied Cryptography is an excellent resource.

0

Share this post


Link to post
Share on other sites
Quote:
Original post by graveyard filla
Do you think you could go more into details about this? Ive looked through a lot of the RakNet source, and it seems very elegant... although why he made all his own data structures is beyond me... What's wrong with RakNet right now that doesn't allow large amounts of players? And could you define 'large'?

Sure thing.

By large I mean that anything over 200 players you are going to start having some major issues, maybe even with a lower amount of connected players.

The main bottleneck in the RakNet API is memory usage for tracking duplicate packets. In ReliabilityLayer.h you will see a giant arry and this is a problem space that needs to be solved as for any large (>100) number of connected clients you are going to have issues. I know Kevin has worked on this but I do not know where he has gotten or what design he chose, and I am pretty sure he does not want to commit until the doxygen and osx port are complete as it would set back other peoples work.

There are other minor issues that really should be cleaned up, and IOCP support is something that you would definately want back in if you are running on a Windows platform server.

Hope that helps.


EDIT:

Also John Schultz I see you mentioning using TCP on specific ports such as HTTP and SSL (80/443). You should be able to use any port without worries as the client is the one opening the TCP connection.
0

Share this post


Link to post
Share on other sites
Quote:
Original post by Saruman
Also John Schultz I see you mentioning using TCP on specific ports such as HTTP and SSL (80/443). You should be able to use any port without worries as the client is the one opening the TCP connection.


That's true, however some environments block everything except HTTP/HTTPS. For example, some users won't be willing to ask the IT department to open ports (security policy may prevent additional port openings).

Quote:
Original post by Saruman
There are other minor issues that really should be cleaned up, and IOCP support is something that you would definately want back in if you are running on a Windows platform server.


Does RakNet use TCP? If not, how do you see IOCP helping a UDP-only based protocol, especially if the server is single-threaded (for maximum performance due to zero (user-level, network) context switching)?

From: http://msdn.microsoft.com/msdnmag/issues/1000/Winsock/

Quote:

Once the completion port has been created and sockets have been associated with it, one or more threads are needed to process the completion notifications. Each thread will sit in a loop that calls GetQueuedCompletionStatus each time through and returns completion notifications.


It would appear that thread context switching overhead might outweigh kernel (paging) advantages with IOCP, especially given the nature of UDP (not using memory/queues as with TCP).
0

Share this post


Link to post
Share on other sites
I could see that as well and would have to investigate the matter further. RakNet is a UDP library and at one point did have IOCP support built into it. Kevin seemed to insist that IOCP was a good thing to have which is why I mentioned it, as I have only used IOCP with TCP based solutions in the past.
0

Share this post


Link to post
Share on other sites
Quote:
Original post by Saruman
I could see that as well and would have to investigate the matter further. RakNet is a UDP library and at one point did have IOCP support built into it. Kevin seemed to insist that IOCP was a good thing to have which is why I mentioned it, as I have only used IOCP with TCP based solutions in the past.


I performed some quick benchmarks with overlapped I/O and UDP, and observed no improvement over simple sockets. I also recall hplus0603 mentioning benchmarking multithreaded UDP (even with MP hardware) and they found single-threaded to be faster. I have not tested overlapped I/O with IOCP, but I suspect that it may be slower than simple sockets for a single-threaded UDP design. Perhaps that's why RakNet dropped IOCP? Benchmark results would be helpful.
0

Share this post


Link to post
Share on other sites
Quote:
Original post by John Schultz
I performed some quick benchmarks with overlapped I/O and UDP, and observed no improvement over simple sockets. I also recall hplus0603 mentioning benchmarking multithreaded UDP (even with MP hardware) and they found single-threaded to be faster. I have not tested overlapped I/O with IOCP, but I suspect that it may be slower than simple sockets for a single-threaded UDP design. Perhaps that's why RakNet dropped IOCP? Benchmark results would be helpful.

The main reason for IOCP being missing was that the feature was broken in a recent version and it just hasn't made its way back into the API as of yet. Also I totally believe that a single-threaded server will have the best performance as you don't get any threads mashing heads.

As I mentioned though this is not the most major bottleneck of RakNet for a large amount of users, the Reliability Layer is the critical design flaw that would need to be adjusted.
0

Share this post


Link to post
Share on other sites
As a general rule the commercial MMO that have had problems with network latency, connection issues and limitations for the number of players being able to play at the same time used TCP. The MMO that have had less problems use UDP. For example compare the problems with WoW that uses TCP with the lack of problems with Everquest that uses UDP.
This is because UDP allows the programmer better control over what data to send and how to send it. Custom UDP protocols are able to compensate for poor network conditions because the application can choose to drop some data to allow the most vital data to get through.
However there has to be a choice for programmers and that is why I programmed ReplicaNet to offer the choice of UDP or TCP for connections. The network transport layer can also be expanded by the user to take advantage of other forms of network communication. For example at least one of the consoles requires socket connections to be authenticated by their online service, so this can be incorporated into a modified transport type.

[Edited by - Martin Piper on May 16, 2005 5:34:33 AM]
0

Share this post


Link to post
Share on other sites
Quote:
Original post by Martin Piper
As a general rule the commercial MMO that have had problems with network latency, connection issues and limitations for the number of players being able to play at the same time used TCP. The MMO that have had less problems use UDP. For example compare the problems with WoW that uses TCP with the lack of problems with Everquest that uses UDP.

Although I agree with what you said about UDP allowing greater control, etc I do not agree with this statement in any way. I can point out 3 MMO games right now that are built on a UDP architecture and are ten times more horrendous latency-wise than any of the TCP counterparts, look how fraught with problems Asherons Call 1&2 have been over their entire lifespan (still broken to this day). Everquest actually had much worse connectivity problems on launch than WoW did, and EQ is even a TCP/UDP hybrid.

I am yet to believe that it is the protocol that causes issues over the actual netcode and implementation of the server.
0

Share this post


Link to post
Share on other sites
Quote:
Everquest actually had much worse connectivity problems on launch than WoW did, and EQ is even a TCP/UDP hybrid.


EQ Only uses UDP for game communication. This can be verified using numerous methods such as capturing the packets or injecting your own code into the winsock dll and logging the socket calls. EQ1 was also much better than WoW at launch in terms of network performance, I know because I was playing both from the start and played WoW from the alpha. :) Everquest2, which uses UDP for game data, also has much better network performance than WoW.

I've also not noticed Asheron's Call problems that are related to them using a specific network protocol. The game might be rubbish, but that is not a network protocol related problem.
0

Share this post


Link to post
Share on other sites
Quote:
Original post by Martin Piper
EQ Only uses UDP for game communication.

Ah ok, I don't know a lot about EQ so I just took the info from the emulator site, thanks for the correction. I confuse this with DAoCs hybrid system I guess.

Quote:

Everquest2, which uses UDP for game data, also has much better network performance than WoW.

With less than 1/4 of the players as well which is a factor.

Quote:

I've also not noticed Asheron's Call problems that are related to them using a specific network protocol. The game might be rubbish, but that is not a network protocol related problem.

My point was that it is not a network protocol related problem exactly. I said that they use UDP but the game runs horribly not because of the protocol but because of the server architecture. Who knows the same could probably be said about WoW.
0

Share this post


Link to post
Share on other sites
I'm still amazed that people even debate this. TCP alone is a poor solution for any kind of a realtime game, if only because even a single dropped packet causes a stall in all network data delivery until that data loss is noted and retransmitted.

Hybrid TCP/UDP systems are needlessly complicated and suffer from problems like bandwidth overconsumption by the TCP stream, maintainance of seperate channels for UDP and TCP, misordered delivery of updates, etc.

I was going to go into greater detail, but then I realized I already have in the design fundamentals section of the Torque Network Library reference. The packet loss section gives a good explanation for why neither UDP nor TCP provide the right abstraction level for realtime game network programming.

- Mark
0

Share this post


Link to post
Share on other sites
Network developers will agree that TCP and TCP+UDP are not optimal for real-time game/simulation applications. Custom UDP protocols can be more efficient. However, if the networked application sends too much data, any advantage provided by a custom UDP protocol is lost. In all cases, if the reliable queues fill up faster than they can drain, lag will be present (beyond network latency), up to the point the channel must be closed due to queue overflow.

A well designed network game can work fine using TCP or TCP+UDP. Beginning network programmers will have a much easier time using TCP+UDP than trying to create a custom UDP protocol from scratch. The argument that TCP+UDP is overly-complex is invalid, as creating a custom UDP protocol is much more complicated. In the context of marketing material for an existing, well-tuned and debugged custom UDP protocol, such an argument can make sense (more in terms of efficiency than complexity: the developer must now become familiar with third party code).

The out-of-order (early) unreliable data arrival before reliable data argument (from the TNL site) is easily dealt with by tracking state. Example: a reliable activation message is sent to an object via the reliable channel and gets lost in transit. An unreliable position update packet arrives before the activation message. The object state is checked, and since the object is not active, the position update is ignored.

How valuable is out-of-order reliable support (OOORS)? In a game/simulation where objects move and stop for long periods of time, or when turning on/off non-state-affecting effects/props, bandwidth can be saved. However, in a game where objects are constantly moving (or stop for very short periods of time), OOORS provides little to no benefit, and if extra packet bits are required to allow for support of OOORS, it's a bandwidth loss.

Quote:
Original post by markf_gg
I'm still amazed that people even debate this. TCP alone is a poor solution for any kind of a realtime game, if only because even a single dropped packet causes a stall in all network data delivery until that data loss is noted and retransmitted.


TCP alone is the only option for games operating in restricted environments (for example, when only HTTP/HTTPS is open at the firewall). As long as the TCP queue is effectively monitored (different methods for *nix and Win32), and decent client-side prediction is implemented, it is possible to work around stall issues.

While TCP will never be ideal for a high-data-rate FPS, it can work fine for RTS and slower moving MMOG's. If the game is primarily running lock-step, where everything must be delivered in order, guaranteed, TCP alone will work fine. During high congestion periods, TCP may, by design, slow down faster than a custom UDP protocol. However, this may be an advantage for a MMOG with thousands of players, where a poorly designed custom UDP protocol may fall apart (keeps sending data at a high(er) rate, preventing the network from recovering). I suspect this is one reason why existing UDP-based MMOG with thousands of players can fall apart. TCP is well designed to efficiently handle this case.

Quote:
Original post by markf_gg
Hybrid TCP/UDP systems are needlessly complicated and suffer from problems like bandwidth overconsumption by the TCP stream, maintainance of seperate channels for UDP and TCP, misordered delivery of updates, etc.


Bandwidth over-consumption is going to come from the unreliable channel, not the reliable channel. During periods of high congestion, the unreliable channel should be cut until the reliable channel(s) queue(s) can drain (data that is not truly state-critical should never be added to the reliable queue). All other arguments can be ameliorated at the network game design layer.

Quote:
Original post by markf_gg
I was going to go into greater detail, but then I realized I already have in the design fundamentals section of the Torque Network Library reference. The packet loss section gives a good explanation for why neither UDP nor TCP provide the right abstraction level for realtime game network programming.
- Mark


The Torque Network Library looks like a good network toolkit (and to be fair, so do RakNet and ReplicaNet). While the arguments given do well to support licensing/purchasing a pre-made, well-tested custom UDP network toolkit, the biggest problem, by far, is network game design as opposed to the underlying network protocol.

I created a custom reliable UDP protocol in a case where the TCP implementation wasn't quite finished. The new UDP protocol ended up being more efficient than a TCP+UDP model (due to packet overhead savings and retransmit optimizations). Even so, the game would grind to a halt during high reliable state data sends. This required a significant redesign of networked game elements. Thus, while every bit of bandwidth helps, the burden of efficiency and game play quality resides in the game design, not the network protocol.

This was for the first full, XBox Live enabled game, and it was finished early (network-enabled games tend to ship late due underestimation of networking issues). While the game only supported 4 players, many more flying and moving objects were active, as well as many rapid reliable state changes (a nature of the game: too late to completely remedy by the time I joined the project). Voice was enabled for all players, all the time (as opposed to only hearing players near each other). The game played with little to no perceptible lag, even below 64kbps (voice took ~32kbps).

Again, I recommend that developers look into developing or licensing/purchasing custom reliable UDP protocols (RakNet, TNL, and ReplicaNet appear to be good choices). However, TCP and TCP+UDP can work fine: the real work in making a game play well under all internet conditions is centered around the network game design itself, not the network protocol. Likewise, if a game plays well/poorly on the internet, it can’t be attributed-to/blamed-on TCP, TCP+UDP, or a custom UDP protocol. It’s the network game design itself.

[Edited by - John Schultz on May 16, 2005 5:55:13 PM]
0

Share this post


Link to post
Share on other sites
Quote:
Original post by graveyard filla
Quote:
Original post by Anonymous Poster
In UDP packets are received in the order that they arrive.


This isn't true. UDP can send packets out of order, and in fact can send duplicates and other nasty things as well.


Yes I should have explained that, sorry. I meant under optimal conditions they are received in the order that they are given, but are certainly not guaranteed to do so. (In fact I did cover this in the other portion of my post, perhaps you didn't read far enough.)

Quote:
In UDP packets are received in the order that they arrive. Which is great for movement, or actions, because the last action is less important than the current action. You can mitigate the problems with out of order sequences on UDP very easily, simply by queuing the messages as they arrive, and then re-requesting those that didn't make it. Once your packets are ordered, then you can process them.


...mitigate the problems with out of order sequences on UDP very easily.....

[Edited by - bit64 on May 16, 2005 5:12:14 PM]
0

Share this post


Link to post
Share on other sites
Quote:
Original post by John Schultz
However, if the networked application sends too much data, any advantage provided by a custom UDP protocol is lost. In all cases, if the reliable queues fill up faster than they can drain, lag will be present (beyond network latency), up to the point the channel must be closed due to queue overflow.

This is one of the reasons why TCP+UDP is a poor combination - it gets people thinking in a message-oriented mindset where the only two primitives for data delivery are guaranteed and unguaranteed messages. What often ends up happening in a 3D simulation is that a large portion of messages get tagged as "reliable", overflowing the queue.

Quote:

How valuable is out-of-order reliable support (OOORS)? In a game/simulation where objects move and stop for long periods of time, or when turning on/off non-state-affecting effects/props, bandwidth can be saved.

In practice we've found out of order delivered reliable events to be of limited use. There is however, no per-packet overhead for supporting reliable OOO data in the TNL model, since TNL also supports strictly unreliable event sends.

Quote:

TCP alone is the only option for games operating in restricted environments (for example, when only HTTP/HTTPS is open at the firewall). As long as the TCP queue is effectively monitored (different methods for *nix and Win32), and decent client-side prediction is implemented, it is possible to work around stall issues.

It should be possible for a network system to support TCP connections for those clients that can't connect via UDP. I think I'll add that as an option to TNL :). It would still support all the higher level primitives like prioritization of object updates and fixed bandwidth consumption.

Quote:

However, this may be an advantage for a MMOG with thousands of players, where a poorly designed custom UDP protocol may fall apart (keeps sending data at a high(er) rate, preventing the network from recovering).

Well, I would never suggest using a poorly designed custom UDP protocol ;)

Quote:

Again, I recommend that developers look into developing or licensing/purchasing custom reliable UDP protocols (RakNet, TNL, and ReplicaNet appear to be good choices). However, TCP and TCP+UDP can work fine: the real work in making a game play well under all internet conditions is centered around the network game design itself, not the network protocol. Likewise, if a game plays well/poorly on the internet, it can’t be attributed-to/blamed-on TCP, TCP+UDP, or a custom UDP protocol. It’s the network game design itself.

The network toolkit you use often affects to a great degree the higher level design of your game networking. TNL, for example, isn't just a low level delivery protocol - it supports a rich set of data delivery policies that greatly simplify the higher level game networking design. TNL also uses a fixed per-client bandwidth setting, meaning that no matter how many objects are being updated, clients network connections will never be flooded.
0

Share this post


Link to post
Share on other sites
Quote:
Original post by Saruman
By large I mean that anything over 200 players you are going to start having some major issues, maybe even with a lower amount of connected players.

The main bottleneck in the RakNet API is memory usage for tracking duplicate packets. In ReliabilityLayer.h you will see a giant arry and this is a problem space that needs to be solved as for any large (>100) number of connected clients you are going to have issues. I know Kevin has worked on this but I do not know where he has gotten or what design he chose, and I am pretty sure he does not want to commit until the doxygen and osx port are complete as it would set back other peoples work.

There are other minor issues that really should be cleaned up, and IOCP support is something that you would definately want back in if you are running on a Windows platform server.

Hope that helps.


wow, thats pretty surprising to me. and to think the architecture in my game that uses RakNet should be able to handle way more then 200 players.... not that i ever expected that many people to play, but it's always nice to be scalable.

0

Share this post


Link to post
Share on other sites
Quote:
Original post by graveyard filla
wow, thats pretty surprising to me. and to think the architecture in my game that uses RakNet should be able to handle way more then 200 players.... not that i ever expected that many people to play, but it's always nice to be scalable.

Note that as I said Kevin will be fixing this so it is not like this will be a persistant issue in the future. You could also fix the main problem yourself just by changing that big array to something more feasible.
0

Share this post


Link to post
Share on other sites
Quote:
Original post by markf_gg
Quote:
Original post by John Schultz
However, if the networked application sends too much data, any advantage provided by a custom UDP protocol is lost. In all cases, if the reliable queues fill up faster than they can drain, lag will be present (beyond network latency), up to the point the channel must be closed due to queue overflow.

This is one of the reasons why TCP+UDP is a poor combination - it gets people thinking in a message-oriented mindset where the only two primitives for data delivery are guaranteed and unguaranteed messages. What often ends up happening in a 3D simulation is that a large portion of messages get tagged as "reliable", overflowing the queue.

Quote:

How valuable is out-of-order reliable support (OOORS)? In a game/simulation where objects move and stop for long periods of time, or when turning on/off non-state-affecting effects/props, bandwidth can be saved.

In practice we've found out of order delivered reliable events to be of limited use. There is however, no per-packet overhead for supporting reliable OOO data in the TNL model, since TNL also supports strictly unreliable event sends.


If OOORS is of limited use, what other class of data beyond guaranteed and non-guaranteed do you see of value? I agree that the biggest problem is too much data sent as guaranteed, which is a network game design issue. I have not yet seen a strong argument for supporting other classes of data. Either the data absolutely has to get there, or it doesn't. Perhaps you can give an example where this is not true?

Quote:
Original post by markf_gg
Quote:

TCP alone is the only option for games operating in restricted environments (for example, when only HTTP/HTTPS is open at the firewall). As long as the TCP queue is effectively monitored (different methods for *nix and Win32), and decent client-side prediction is implemented, it is possible to work around stall issues.

It should be possible for a network system to support TCP connections for those clients that can't connect via UDP. I think I'll add that as an option to TNL :). It would still support all the higher level primitives like prioritization of object updates and fixed bandwidth consumption.


That's cool. When you get it working, perhaps post benchmarks showing any performance differences between the two (for a variety of bandwidth and network conditions, game types, etc.). I believe you'll need to use overlapped I/O and IOCP to determine the TCP send queue state on Win32 (queue flags/read-options exist for *nix).

Quote:
Original post by markf_gg
Quote:

However, this may be an advantage for a MMOG with thousands of players, where a poorly designed custom UDP protocol may fall apart (keeps sending data at a high(er) rate, preventing the network from recovering).

Well, I would never suggest using a poorly designed custom UDP protocol ;)


The developer may not know that their implementation is poor until stressed under real-world internet conditions. WRT the previous link, I discovered that my first custom protocol was quite poor during network simulation and analysis (I thought it was decent before testing). It was during this analysis that I found that even the native TCP implementation (for this device) was broken. This is one advantage to using a pre-made toolkit, provided the toolkit authors can provide benchmarks/statistics showing that their design can survive worst-case internet conditions (thousands of players, etc., as with an MMOG). TCP has been researched/studied for around 20 years: it's strengths and weaknesses are well known (internet bandwidth balancing/optimization is still a hard, unsolved problem (not perfected)). RED and WRED are newer designs for router queues that help TCP to behave more efficiently during high congestion situations. This an another reason to start with the basic TCP design when designing a custom protocol.

Quote:
Original post by markf_gg
Quote:

Again, I recommend that developers look into developing or licensing/purchasing custom reliable UDP protocols (RakNet, TNL, and ReplicaNet appear to be good choices). However, TCP and TCP+UDP can work fine: the real work in making a game play well under all internet conditions is centered around the network game design itself, not the network protocol. Likewise, if a game plays well/poorly on the internet, it can't be attributed-to/blamed-on TCP, TCP+UDP, or a custom UDP protocol. It’s the network game design itself.

The network toolkit you use often affects to a great degree the higher level design of your game networking. TNL, for example, isn't just a low level delivery protocol - it supports a rich set of data delivery policies that greatly simplify the higher level game networking design. TNL also uses a fixed per-client bandwidth setting, meaning that no matter how many objects are being updated, clients network connections will never be flooded.


That's true. My point has been that arguments against TCP and TCP-UDP are without merit in the cases where developers are aware of the network game design issues and are (for whatever reason: time, policy, skill level, cost, target market) limited to a TCP or TCP-UDP solution. In cases of congestion, TNL must drop any clients that haven't allowed the server to drain its reliable queue(s): at some point, no matter how much queue memory is present, you've got to call it quits, and drop the client. While you state that TNL supports a fixed bandwidth option, do you mean fixed maximum bandwidth? Does TNL also implement a filtered mean-deviation estimator to dynamically (and near optimally) adjust bandwidth for live internet conditions? The latter is far more important than the former (I would look at the former as a tuning tool, but in general would not want to artificially limit client bandwidth: there is no reason to do so if the custom protocol is optimally adapting for all the live channels).
0

Share this post


Link to post
Share on other sites
Quote:
Original post by John Schultz
If OOORS is of limited use, what other class of data beyond guaranteed and non-guaranteed do you see of value? I agree that the biggest problem is too much data sent as guaranteed, which is a network game design issue. I have not yet seen a strong argument for supporting other classes of data. Either the data absolutely has to get there, or it doesn't. Perhaps you can give an example where this is not true?

TNL and the Torque and Tribes engines before it introduced a data delivery policy called "most recent state guarantee" which is to say that for a given object, the current state of the object will, at some point, be reflected to clients interested in that object. This is at the heart of the ghosting facility of TNL and what sets it apart from many other networking packages (i.e. RakNet).

In this system, rather than having simulation objects "push" data events to clients, the objects simply mark themselves as having dirty states. When TNL decides it's time to send another packet to a particular client, it sorts dirty objects based on a user-supplied prioritization function and then writes object updates into the packet until the packet is full. Any dropped packets simply set the dirty state flags for that object for that client that were not subsequently updated in a later packet.

All remote object (ghost) creation messages are sent using this system as well, thus substantially limiting both the number of guaranteed and unguaranteed messages sent to clients. Because TNL tracks which states exist on which clients, there's no need to pulse unguaranteed messages (in case some position state was lost) or to send lots of guaranteed object creation/deletion messages.

The other data classificiation I've found useful is the "quickest delivery" data type -- player input for example, where a dropped packet or two shouldn't require a round-trip back to the client for a re-send. This is mainly a presentation issue for other clients in the simulation.
Quote:

The developer may not know that their implementation is poor until stressed under real-world internet conditions.

Well, they could always use a network technology that's been proven successful in AAA networked games back to oh, say 1998...

Quote:

While you state that TNL supports a fixed bandwidth option, do you mean fixed maximum bandwidth? Does TNL also implement a filtered mean-deviation estimator to dynamically (and near optimally) adjust bandwidth for live internet conditions?

TNL does have an adaptive bandwidth option for connections, but it's fairly primitive at this point. In the products I've shipped with it (Starsiege: TRIBES and Tribes 2) we simply fixed the client bandwith at 2kbytes/sec and 3kbytes/sec respectively. Due to the nature of the most-recent state data guarantee we always filled up each packet to the client. The resulting gameplay was of sufficient quality that we didn't bother attempting to adaptively adjust bandwidth settings on the fly, although we did allow clients to adjust the params slightly upwards if they had broadband connections.

I am currently looking at improving our adaptive rate code to more easily allow TNL's use in higher bandwidth, non-simulation applications. Can you recommend anything I should read on the subject? A near optimal filtered mean-deviation estimator sounds like it might be what I'm looking for :)
0

Share this post


Link to post
Share on other sites
Quote:
Original post by markf_gg
Quote:
Original post by John Schultz
If OOORS is of limited use, what other class of data beyond guaranteed and non-guaranteed do you see of value? I agree that the biggest problem is too much data sent as guaranteed, which is a network game design issue. I have not yet seen a strong argument for supporting other classes of data. Either the data absolutely has to get there, or it doesn't. Perhaps you can give an example where this is not true?

In this system, rather than having simulation objects "push" data events to clients, the objects simply mark themselves as having dirty states. When TNL decides it's time to send another packet to a particular client, it sorts dirty objects based on a user-supplied prioritization function and then writes object updates into the packet until the packet is full. Any dropped packets simply set the dirty state flags for that object for that client that were not subsequently updated in a later packet.


This makes sense for objects that are rapidly changing state when the system is not running lock-step (or don't require ordered state consistency). To date, I have not run into a problem where this method can provide significant bandwidth savings, but I'll keep it in mind as a future option.

Quote:
Original post by markf_gg
Quote:

The developer may not know that their implementation is poor until stressed under real-world internet conditions.

Well, they could always use a network technology that's been proven successful in AAA networked games back to oh, say 1998...


Given that this thread is titled MMORPG..., have you tested TNL with 2000-3000 player connections, under real-world internet conditions?

Quote:
Original post by markf_gg
Quote:

While you state that TNL supports a fixed bandwidth option, do you mean fixed maximum bandwidth? Does TNL also implement a filtered mean-deviation estimator to dynamically (and near optimally) adjust bandwidth for live internet conditions?

TNL does have an adaptive bandwidth option for connections, but it's fairly primitive at this point. In the products I've shipped with it (Starsiege: TRIBES and Tribes 2) we simply fixed the client bandwith at 2kbytes/sec and 3kbytes/sec respectively. Due to the nature of the most-recent state data guarantee we always filled up each packet to the client. The resulting gameplay was of sufficient quality that we didn't bother attempting to adaptively adjust bandwidth settings on the fly, although we did allow clients to adjust the params slightly upwards if they had broadband connections.


Worst case scenario analysis for a MMORPG and 3000 very active players:

3kbytes/sec * 3000 players = 9000kbytes/sec, 72,000kbits/sec, 72Mbits/sec.

This means you'll probably have many fat pipes, as well as extra routing capabilities to deal with varying internet conditions. Given the unpredictability of network conditions, if the server does not actively adapt its bandwidth output, the system is going to fall apart (lots of data, lots of connections, lots of unpredictability). While this example isn't much of a proof, the bandwidth/complexity concepts come from studying the design of TCP, and why it is able to allow millions (billions) of connections to run relatively smoothly over a very complicated network (or web) of dataflows.

Quote:
Original post by markf_gg
I am currently looking at improving our adaptive rate code to more easily allow TNL's use in higher bandwidth, non-simulation applications. Can you recommend anything I should read on the subject? A near optimal filtered mean-deviation estimator sounds like it might be what I'm looking for :)


Van Jacobson's paper, Congestion Avoidance and Control, written in 1988, is an excellent starting point. The history is also fascinating: it describes a time when the early internet could collapse. In the almost 20 years since the paper was written, there have not been signficant improvements (for all cases). These algorithms, as well as their variants, make up the core features of, you guessed it, TCP. To go back further in time, see RFC793, written for DARPA in 1981.

Thus, I hope it is clear why I have been defending TCP*: it really is an excellent protocol. Some of its features are not ideal for games/simulations, but the RTO calculation (see appendix A in Jacobson's paper) is required if a custom protocol is to be used in a large-scale, real world internet environment (such as a MMORPG). It's probable that the UDP-based MMORPG's that fail/fall-apart is due to poor bandwidth management.

More links here.

In summary, study the history and design of TCP, and use the best feature(s) for custom game/simulation protocols, while leaving out (or retuning) features that hurt game/simulation performance.




* I believe this is the first paper to describe TCP, by Vinton Cerf and Robert Kahn in 1974, BSW (Before Star Wars ;-)). TCP/IP allowed ARPANET to become the Internet and later the World-Wide Web. Robert Kahn talks about TCP and the birth of UDP.

[Edited by - John Schultz on May 18, 2005 4:19:23 AM]
0

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this  
Followers 0