Sign in to follow this  
wmarsh

Expertise from network games programmers required

Recommended Posts

...to settle a bit of an Internet bust-up. The conversation between "TobiHeidi" and myself, "wm": https://www.mochiads.com/community/forum/topic/realtime-multiplayer-flash-game#29 Thoughts? I'm not looking for backup for my statements, but neutral thoughts from people who program networked games day in, day out.

Share this post


Link to post
Share on other sites
What is there to discuss?

I think it's clear that TobiHeidi has no idea what he's talking about. He claims Counter-Strike uses P2P (which it doesn't) and that packet loss is 'very rare' in the wild (ie not your typical LAN) which isn't true either.

Edit: Doh, you're WM - I didn't understand that, hehe. Well, using any protocol on a lossy Wireless connection (which isn't that unrealistic to expect) will yield huge amounts of spiky losses of packets, and ADSL and modems are not exactly immune to it either.

Don't feed the trolls, I'd say. :)

Share this post


Link to post
Share on other sites
I have a real-time server (C# with IOCP sockets) that's I've tested with 100 flash clients. I'm not sure where you got the idea that you can't use TCP. You might be assuming packet loss is high and that dropped packets happen so much that it's a problem. I haven't seen this myself even with testing with people across the world. He's right about the once a month stuff. I honestly never see my RTT jumping. It's normally constant for each client.

You have to be careful with TCP connections. A lot of beginners will flood the socket. It's a good idea to run a fixed update. I use 100 ms physics updates and I throttle packets between 5 and 10 packets/second. If you come from a UDP background you might be trying to shove 30 packets down the line, but don't. Naggle the packets yourself and send them messages in batches like I said.

Having helped a few people online with the idea of flash networking I found out a lot of them had misconceptions or completely set up things wrong and were asking for trouble.

// edit I have to go home. I'll finish reading your posts later.

Share this post


Link to post
Share on other sites
Is this ideal for something that needs to be very low latency like a space shooter or FPS game though Sirisian? I just find it hard to believe that anything like this wouldn't be drastically improved by being able to throw out UDP packets as quickly as possible.

I can't agree with what you say about lost packets. Have you tested in the real world where people have line noise and crappy wireless connections and congested networks etc.?

Share this post


Link to post
Share on other sites
Quote:
Original post by wmarsh
Is this ideal for something that needs to be very low latency like a space shooter or FPS game though Sirisian? I just find it hard to believe that anything like this wouldn't be drastically improved by being able to throw out UDP packets as quickly as possible.


Unless you can advance simulation despite missing packets, there is no inherent benefit in using UDP.

Quick has nothing to do with real time. Even with UDP, sending packets over flooded connection will make things worse. TCP in this case provides throttling mechanisms, with UDP handling that is up to user.

Real time is about meeting a deadline. In case of games, this means that by the time simulation needs to advance to next step, it has received enough information (not data) to be able to do something meaningful about remote state. If that means skipping missing states, advancing based on prediction, waiting for data, or something else is purely up to what fits the requirements.

The key advantage over using UDP is the ability to define custom recovery mechanism, or not recover at all, instead of using the standardized one implemented by TCP.

Although, it might not even matter, since many connections don't even support it, or tunnel it through other types of connections, especially with increased emphasis on mobile platforms (G3 and related phone connections) and browser (AJAX, and similar).

Most of the time it is considerably more important to mask the effects of latency. For example, most of Windows 7 is not about performance improvements, but about masking Vista's perceived sluggishness using various UI tricks.

Share this post


Link to post
Share on other sites
My take on it is that if data has arrived corrupted, I no longer want that data. It's too late, I want to move on. I don't want a system that keeps trying to send a packet from the past until I get it successfully, instead I want to act on whatever packets do arrive after that.

I would rather have a client keep predicting the action until it does get a valid update from the server than wait for the network to provide me with an old, correct packet. I understand that none of this happens automatically because I've chosen to use UDP, and that of course the client and the server must define their own ways of dealing with network events like this, but that's what I want!

This is my point really, and I'm sure I've failed to make it clearly: It's not a case of "Ooh, UDP is 20ms faster than TCP! Use that by default", but instead a case of not wanting to use TCP's methods of correction and timing and delivery because they are not suitable for what I'm doing. Looking at what has been happening in online gaming since it has been relevant, I believe that this is the general take on the matter.

Share this post


Link to post
Share on other sites
Quote:
Original post by wmarsh
My take on it is that if data has arrived corrupted, I no longer want that data. It's too late, I want to move on. I don't want a system that keeps trying to send a packet from the past until I get it successfully, instead I want to act on whatever packets do arrive after that.


If you can do that, sure, why not use the optimal method.

There is however another side. Developing a reliable UDP transport is difficult(tm). So usually either third-party library is used, developers do their best, or "better" way is used (TCP). Practical experience shows that often, using inferior, but more accessible choice will result in superior end results, even if sub-optimal.

I like this. It is perhaps the true representation of software development in general. Technical issues are the last of the problem, and easiest to solve.

But as short answer - if you have the choice, the expertise, the time, and all that, then rolling custom UDP networking mechanism may result in better user experience. The word 'may' does not imply that using TCP by itself would be better. For example, home routers can be a disaster for UDP, moreso than TCP.

And since testing and QA may be simply too expensive for the developer(s), they can opt for a tried-and-true inferior solution with better end results.

Share this post


Link to post
Share on other sites
Quote:
Original post by wmarsh
This is my point really, and I'm sure I've failed to make it clearly: It's not a case of "Ooh, UDP is 20ms faster than TCP! Use that by default", but instead a case of not wanting to use TCP's methods of correction and timing and delivery because they are not suitable for what I'm doing. Looking at what has been happening in online gaming since it has been relevant, I believe that this is the general take on the matter.


Yeah, at least that's how they do it in Unreal Tournament and the Tribes II papers (by Frohmayer) describe the same procedure.

It's possible that a packet isn't delivered and meanwhile the simulation has advanced and retransmitting the packet is pointless because it contains data that is out of date. As far as I understand, this isn't possible to do in TCP.

I guess that's also why most twitch games use UDP.

Share this post


Link to post
Share on other sites
UDP has advantages as you've pointed out. The major one you pointed out is being able to decide which packets are ordered or reliable and setting things up for optimal performance.

The Source networking is explained here

It would also depend on the type of gameplay in the FPS. If it's slower paced then TCP would probably be fine. UDP works really well for FPS games so there's not much of an incentive for professional developers with network programmers to choose TCP.

(oh and yes my game is real-time. I was working on a physics engine, but I got side tracked with something else. The networking is highly optimized though and built to be "playable" at 2 packets/second and such. 5 packets/second makes it easily playable and responsive. It's a top down shooter so 200 ms doesn't feel too bad. 10 might be overkill but if a player's latency is <100 ms I go that high.)

Share this post


Link to post
Share on other sites
Quote:
Original post by Sirisian
UDP has advantages as you've pointed out. The major one you pointed out is being able to decide which packets are ordered or reliable and setting things up for optimal performance.

The Source networking is explained here


This is not UDP anymore than TCP is. Both, TCP and Source protocol, send packets. The difference comes from how they send them.

This is similar to:
1. Use UDP
2. ???
3. Profit

Obviously UDP allows more domain-specific network utilization. But it's the 2. that makes the difference.

Share this post


Link to post
Share on other sites
I didn't mean to make any connection between UDP and the Source networking. I just realized the OP brought up CS:S so I thought I'd paste that link.

Share this post


Link to post
Share on other sites
Quote:
Original post by Sirisian
I didn't mean to make any connection between UDP and the Source networking. I just realized the OP brought up CS:S so I thought I'd paste that link.


I know. But in such comparisons it's always about UDP (+ something unknown) vs. TCP. Custom-tailored protocols can be incredibly efficient, but can also be a PITA.

You've mentioned you helped others debug their TCP applications. That is one benefit of standardization, which would not be possible if they had their hand-rolled protocol on top of UDP, at least not without considerable individual effort. The task then becomes not only fixing the application, but often also the underlying layers.

Share this post


Link to post
Share on other sites
Right, thank you all. That's great. I now feel comfortable and adjusted in my assumptions about TCP and UDP!

I am surprised and intrigued that your game does run quickly with TCP Sirisian. Is it (as SymLinked describes) a "twitch" game or something less hair-trigger oriented that can get away with masking the latency?

It's good to see that Source is a client-server, and the article explicitly stating that there is no peer-to-peer stuff going on at all. TobiHeidi's assertion that lack of P2P constituted the bottleneck was making me question reality. Fair enough if he feels the same way about my UDP claim!

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Quote:
Original post by Sirisian
UDP has advantages as you've pointed out. The major one you pointed out is being able to decide which packets are ordered or reliable and setting things up for optimal performance.

The Source networking is explained here


This is not UDP anymore than TCP is. Both, TCP and Source protocol, send packets. The difference comes from how they send them.

This is similar to:
1. Use UDP
2. ???
3. Profit

Obviously UDP allows more domain-specific network utilization. But it's the 2. that makes the difference.
Yes, but if you are going to design the best game networking library you can with this design in mind, of course you're not going to choose to build it on top of the protocol that replicates half of the functionality you are building yourself. I don't really see how being aware of that decision is equivalent to thinking that UDP is a magic speed wand.

Share this post


Link to post
Share on other sites
Quote:
Original post by wmarsh

I don't really see how being aware of that decision is equivalent to thinking that UDP is a magic speed wand.


My point is - building your own protocol over UDP can result in worse behavior than that of TCP. I can't remember which game or application would consistently crash one of my home routers when it tried to resolve connection problems by flooding the connection with dozens or even hundreds of packets (working as designed, according to tech support).

There are also various security issues in both, protocol design as well as implementation, which TCP has included over the years, as well as various empirical observations with regard to network traffic. None of those ever included strictly real-time requirements, and as such it isn't best fit.

This is why defining that "best" is much more important than just starting off with barebone UDP. While TCP often isn't ideal (yet can frequently be good enough), other UDP-based libraries might already meet the requirements, but they cannot be called UDP.

Share this post


Link to post
Share on other sites
Quote:
Original post by wmarsh
I am surprised and intrigued that your game does run quickly with TCP Sirisian. Is it (as SymLinked describes) a "twitch" game or something less hair-trigger oriented that can get away with masking the latency?


Not an answer to your question directly, but I'm going to throw around some default values here.

Tribes II:
Client -> Server = 8 packets per second.
Server -> Client = 32 packets per second.
The game ticks at 32 mhz.

Counter-Strike:
Client -> Server = Not sure.
Server -> Client = 32 - 100 packets per second.
Like the network update rate, the game can tick from 32 mhz to 100 mhz depending on the server. It wasn't rare to see tournament servers using higher values but as far as I remember ~60 mhz was the norm.

Also consider that Tribes II doesn't mask shooting latency, while CSS does - where when you fire the trigger the fire animation is played on the client immediatly. Both do mask movement latency however.

Share this post


Link to post
Share on other sites
Thanks for the thoughts and info.

If I could just throw something else into the mix: back in the context of Flash, we could be talking about some very simple games. Some examples of multiplayer Flash games that I have seen are "fridge magnet" games, where you re-arrange letters on a big board and spell out silly words with your co-players (okay, not really a game) or racing games, in which the networking component can (in the context of silly Flash applets) be as simple as telling the server your currrent position, and graphically plotting the positions of other racers (which you don't even interact with). Apart from that there's a bit of simplistic control at the end of rounds etc. to report who won and a lobby system to arrange new races.

With that simplicity in mind you could get away with very simply sending your position from the client and receiving the position of other players from the server. Anything else be damned, you can just treat these packets as instant status updates and nothing more complex. [url=http://jsw.ovine.net/]Jet Set Willy Online[/url] does this. Isn't UDP a positive boon in this case? I don't even care about broken packets, it's a Flash applet. At worst I'll see another player in the wrong place briefly. You only need to be careful during the lobby/end of race bits, and that's not realtime gameplay so that's easy too.

Share this post


Link to post
Share on other sites
Quote:
Original post by wmarsh
I am surprised and intrigued that your game does run quickly with TCP Sirisian. Is it (as SymLinked describes) a "twitch" game or something less hair-trigger oriented that can get away with masking the latency?
Having the client run the simulation in sync with the server works fairly well. For players I have a very slight inertia implemented. Last time I tested I just had something similar to linear extrapolation implemented which seemed to work well. It's not FPS fast. I limit rotation speed and extrapolate and interpolate that also.

A lot of my time was invested in input synchronization. A lot of what I'm going to say is completely obvious probably. My input system stores all input from the client and pushes it into a queue. This input includes mouse positions (yes I have the client send mouse positions) along with key and mouse (I use the javascript hack to turn on the right mouse button) up and down states with timestamps. The client runs it's packet send at 33 ms updates (For testing I was using 200 ms it looks like). At this time it runs a compression algorithm. (It's in binary, but I'll write it in text) A time is selected like 200 ms for mouse and 100 for keys or something.

// Last packet sent at 30000
LeftMouseDown 30020 ms
LeftMouseUp 30150 ms
KeyDown A 30170 ms
KeyDown W 30195 ms
RightMouseDown 30190 ms
// 30200
KeyUp A 30220 ms
RightMouseUp 30250 ms
// 30400

Then lets say we decide to send a packet at 30200 aka 200 ms difference. The first two items compress into a LeftMouseClick. The key down for A is detected. It scans ahead to see if a a key up is in the queue but at 30200 ms no key up has come. So it delays that input and delays the next key up and the right mouse down. So only the LeftMouseClick is sent. In the next packet it starts at the keydown for A and scans ahead and finds that an up occurs in < 100 ms interval and compressed it into "KeyPress A". Then it looks at keydown W and when it scans ahead no up has been detected in <100 ms. So it adds "KeyDown W" to the outgoing quue. The right mouse is compressed to a click. All the data is then sent. All the mouse events have coordinates sent with them.

That's about it. I haven't worked on it in a while. I never tested by having someone far away hosting. I just trusted people in australia and such to tell me of the game felt unresponsive. It tells the player the latency and bounces back all player interaction to all the connected clients for fun. So everyone can each others mouse and clicking and other events. I've heard it ran well. Since I'm hosting my server in debug most of the time I don't feel the full RTT effect.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this