Sign in to follow this  

Networking protocol via UDP for an MMO game

This topic is 405 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey,

 

I know there are lot of topics about UDP vs. TCP for real-time games, but I'm asking for a different thing.

 

Assuming, I've decided to use UDP on low-level.

Is there any existed messaging/packets protocol for a multiplayer networking or specifically for replication and sync?

Or each game developer is making own protocol from a scratch each time?

 

Asking, because those basic networking things are very common for any multiplayer game.

At least, the replication issue is for sure exists for any multiplayer game and especially for MMO games.

 

And I'm asking not about exact implementation, rather about a description/specs for an algorithm which will be fast and reliable.

 

Probably, there should be some projects about this, since MMO and multiplayer games are pretty common.

 

I completely understand, that each game have own specific requirements, but basic stuff is the same, or, at least, very similar – I'm talking about replication, players synchronization, prediction/interpolation and so on.

 

If you think it's not possible to develop a common protocol for different games – please, could you explain why exactly it's not possible? (besides basic "just all games are completely different" – as I wrote above, it's not true: there are lot of common things in terms of networking)

Share this post


Link to post
Share on other sites

Is there any existed messaging/packets protocol for a multiplayer networking or specifically for replication and sync?
Or each game developer is making own protocol from a scratch each time?


There are no "standard protocols" similar to how there might be RTP or NTP or NFS or whatever going over UDP.
The reason for this is twofold:
1. There is no need for interoperability -- the developers of Overwatch aren't interested in talking to a running instance of Counter-Strike.
2. The choices made in networking implementation are very important to the feel of fast-action games. What works for Unreal Tournament may be terrible for Doom.

There exists various libraries that "solve this problem" for game developers, if you can live with the assumptions made by those libraries.
This solves the commercial pressure that "if I can use something off the shelf, I can save development time."
Multiple games that use the same libraries will not be able to talk to each other, though, as the gameplay code is totally different.
Examples of libraries includes built-in networking in engines such as Unreal Engine or Source (these make all the choices for you) as well as less opinionated libraries like Lidgren, RakNet, and ENet.

Even if, for some games, interoperability "could" be built, there is zero commercial value in doing so, and thus it isn't built.
For the case of virtual worlds, where some amount of interoperability could perhaps actually make sense to users, I attempted to get the IETF to adopt an approach that would provide at least limited interoperability, but then 2008 happened and nobody had money to worry about that anymore.
https://tools.ietf.org/html/draft-jwatte-less-protocol-01

There are about four basic approaches that actually work. Those approaches are reasonably well known, but no one "bible" book describes them all.
The first approach is "spray and pray" -- just send commands as they happen from the client, and states as they update from the server, and let interpolation sort it out. Very simple, and can be effective for certain kinds of games. Drawback: Poor handling of player/player interactions.
The second approach is "lockstep synchronization" -- send commands for time X ahead of time to the server, which forwards to everyone else; the time step X isn't executed on all machines until they have all participants' commands for time X, then all the commands are executed in lock step; no actual state needs to be replicated as long as simulation is deterministic. Drawback: Long command latency before you see the action on the screen.
The third approach is "client ahead, remote entities behind" -- send commands for time X ahead of time to the server, and simulate them immediately on the client. Drawback: Remote entities are in a different place relative to you on your machine, compared to the server.
The fourth approach is "client ahead, remote entities forward extrapolated" -- same as the previous one, but forward extrapolate remote entities to approximate locations where they would be in the future. Drawback: May display remote entities in positions they will never be in, because they turn around instead of going forward or whatever. Edited by hplus0603

Share this post


Link to post
Share on other sites
There are about four basic approaches that actually work. Those approaches are reasonably well known, but no one "bible" book describes them all.

The first approach is "spray and pray" -- just send commands as they happen from the client, and states as they update from the server, and let interpolation sort it out. Very simple, and can be effective for certain kinds of games. Drawback: Poor handling of player/player interactions.

The second approach is "lockstep synchronization" -- send commands for time X ahead of time to the server, which forwards to everyone else; the time step X isn't executed on all machines until they have all participants' commands for time X, then all the commands are executed in lock step; no actual state needs to be replicated as long as simulation is deterministic. Drawback: Long command latency before you see the action on the screen.

The third approach is "client ahead, remote entities behind" -- send commands for time X ahead of time to the server, and simulate them immediately on the client. Drawback: Remote entities are in a different place relative to you on your machine, compared to the server.

The fourth approach is "client ahead, remote entities forward extrapolated" -- same as the previous one, but forward extrapolate remote entities to approximate locations where they would be in the future. Drawback: May display remote entities in positions they will never be in, because they turn around instead of going forward or whatever.

Thanks you, this is exactly what I've asked for, just I've hoped there could be a strict specs for such things... but now I see it's unlikely, so will have to implement it from scratch and spend time to debug and so on and so on... That's why I don't really understand why there are no common protocols for such things – it could save a lot of gamedev time :-)

Edited by norlin

Share this post


Link to post
Share on other sites

That's why I don't really understand why there are no common protocols for such things – it could save a lot of gamedev time :-)


But nobody would make money from that protocol being public, because there is no incentive or need for two different games to talk to each other.
Also, given that the existing systems already have working implementations, it's not in their interest to document the protocols at a low level, because that would make life easier for possible competitors.

If you just want to save time, you should use an existing engine! Download Unreal Engine and use its system; it's already implemented and works.

Share this post


Link to post
Share on other sites

to talk to each other.
No need "to talk to each other", I'm just talking about re-using existing, already tested things for a most common part of any multiplayer game.

But you're right it's not a profit for those who developing such protocol. But it's a profit for a whole gamedev community, as any open-source things does. What could we made with Internet, if anyone develop own HTTP-like protocol for each site? (I understand it's not quite equal comparison, but still...)

 

Download Unreal Engine and use its system
Huh, I'm exactly learning UE4 for now, and all their networking features are hidden and not documented. At the same time, UE4 for server-side requires a windows, at least for compiling, which seems VERY strang for me in case of cross-platform engine...

 

Basically, I'm also looking for a hybrid way – to use UE4 networking and move all other server-side logic to my custom solutions, but it's an offtopic here.

Share this post


Link to post
Share on other sites

The most "finished" library I know that is used in games is RakNet, I personaly prefered to roll my own simple just API level network service for TCP/UDP that works in Windows and Linux/Mac, Android using Socket and IOCP for few suers and massive users at the same time. Anything else depends on your game architecture. Everybody in every studio has its own prefered way to do and this may be also the reason why there isnt a general solution as you might seek for.

 

You need to decide what the server does, what the client does and how they work together. In most MMOs server does the gameplay and only tells client for the results, in other games there might be most processing power on the client side and server just evaluates stuff only and in some games server just spreads update messages only (Like in Dungeons and Dragons Daggerdale) what might result in asynchronously game states on each client where one client has had the intro already but another is still watching it, shops have different items for each player and such things

Share this post


Link to post
Share on other sites

No need "to talk to each other", I'm just talking about re-using existing, already tested things for a most common part of any multiplayer game. But you're right it's not a profit for those who developing such protocol. But it's a profit for a whole gamedev community, as any open-source things does.
Not necessarily. It's hard to make a shoe that fits on every foot, but for MMO (note that there are two Ms, not just one) it is twice hard. People write their own because it seems plausible that they can squeeze out a few more bits compared to someone who writes code ahead of time without even knowing any detail about your game. Bandwdth costs money.

 

That being said, there exist open, well-tested libraries, and they are being used (Raknet has already been mentioned, enet would be another much less feature-rich example). Or, some just use TCP. Even that can work well (it really depends). Early versions of one particular successful MMO used unencrypted XML-RPC over TCP in the early 2000s. Yep, that worked fine for up to 1,500 connections per instance, don't ask me how they made it work!

 

Also, note that talking to the client isn't everything in a MMO. Usually (basically, always), the outward-facing machine(s) talking to clients are not the same phyiscal machines as the ones running the simulation, or the ones storing data, and often/usually "login" and "gameplay" are different machines, too. These servers need to talk with each other in some way. You might very well use something like zeromq/nanomsg for inter-server communication, for example.

Share this post


Link to post
Share on other sites

In my time working on MMOs I've always been impressed at (a) how different the networking on each of them was, and (b) how much that was down to personal choice rather than accident.

 

For general object replication, my own code used a system much like the LESS protocol above, sent over TCP because it wasn't a fast-paced game. However one previous project completely refused to have automatically serialised properties because they wanted all updates to be explicit, and because their multi-server model required it. A 3rd project stuck to the ancient "one struct per message type, manually serialised" method because that's what they were comfortable with, and because their codebase was mature enough that they already had most of the message types they needed.

 

The prediction and interpolation layer was different on all three projects too, ranging from fully "client ahead, remote entities behind" on my code to extensive extrapolation and correction on another because they had to accommodate shooter-style gameplay with the same engine.

 

So actually, the stuff that the OP suggested was "the same, or, at least, very similar" was probably the area in which the networking differed the most.

Share this post


Link to post
Share on other sites

I'm completely agreed that everyone has to implement own custom solutions for now. But it does not looks right way, since the issues are exactly the same for almost any game which you've described (I don't talking about personal preferences, just about technical issues).

 

Of course, in case if game has solid code base, it will not fit to any common protocol, but for a new game – I don't see a real reason why need to invent networking from the scratch every time.

Again, websites are all different, using multiple requests types, receiving and sending completely different data and so on, but all sites are using the same HTTP protocol.

 

So, for games, what's required, basically: low latency, optional errors handling, prediction/interpolation – and all this could be exactly the same for most of multiplayer games (not only MMO), technically.

 

You're talking that each game has made own solution – well, it's obvious, since we don't have a public one. They just don't have another choice.

 

I have pretty few experience with multiplayer networking, but maybe you can give me 2 examples of multiplayer games, which will be controversial to each other in terms of networking? (I'm talking not about existing games, rather about 2 design examples).

Share this post


Link to post
Share on other sites

the issues are exactly the same for almost any game which you've described (I don't talking about personal preferences, just about technical issues).

 

No, they're not. The choices made regarding the game design and the technical constraints go hand in hand. Twitchy-shooters don't want to run over TCP with other clients shown at old locations. Slower RPG-type games will be fine with TCP and will prefer smooth movement of other clients with no corrections needed. Games where all you do is move and shoot will be highly optimised for low latency and resilient to dropped packets whereas games where the interactions with the world are more complex may be optimised for arbitrary messages and RPC mechanisms where everything is expected to be sent reliably and in-order.

 

Even 2 shooters can have different philosophies on how to handle interpolation or extrapolation. Some games focus on trying to be as perfectly synchronised as possible, at the expense of needing to perform more corrections. This might work well for eSports. Others happily rewind game state so that someone who already reached cover can be shot by someone who pulled the trigger when the target wasn't in cover yet, because the engine prefers a consistent experience for the shooter rather than the shootee.

 

You keep using HTTP as an example but it is a poor comparison. HTTP is designed as a standard for interoperability, not as a way to accelerate the development of web servers and clients. You choose to make a website that is served over HTTP because you want people with existing clients to access it, and because you can cope with the limitations of HTTP. And if you can't, that's why we have websockets, FTP, etc.

Share this post


Link to post
Share on other sites

I have pretty few experience with multiplayer networking, but maybe you can give me 2 examples of multiplayer games, which will be controversial to each other in terms of networking? (I'm talking not about existing games, rather about 2 design examples)
Compare Overwatch (6 vs 6) or Titanfall, Battlefield, or anything similar to Anarchy Online (hundred vs hundred, and tens of thousands in the same world) or World of Warcraft (few vs few, but tens of thousands in the same world) on the other side. Compare any of these to any game similar to... Forge of Empires, or to any of the countless "build your own village" games.

 

Compare in each case what the total number of players are, what the numbers are in your immediate visible neighbourhood, compare how far you can see and how you can interact, and compare how responsive the game must be. Also compare how "realtime" each of these games really are (hint: time does not always pass at the same pace in each of the forementioned).

 

To give an example why your design in any of these may differ greatly: Half a second of lag isn't great in a strategy or "build a town" game, but it's acceptable. In a "typical" MMO, you'll have complaints on your user forums or people leaving, and in a shooter game, you will receive massive shit storms and death threats. In a round-based game (which includes pretty much every board/card game), half a second isn't even noticeable.

Share this post


Link to post
Share on other sites

Compare...
So, I know the games are different. But which conflicts do you see from networking? We can't take slow protocol for a fast-paced game, but we can easily take fast protocol for a slow game. And in case if such protocol already exist, even slow games could use it with no costs for development from this side.

 

To give an example why your design in any of these may differ greatly: Half a second of lag isn't great in a strategy or "build a town" game, but it's acceptable. In a "typical" MMO, you'll have complaints on your user forums or people leaving, and in a shooter game, you will receive massive shit storms and death threats. In a round-based game (which includes pretty much every board/card game), half a second isn't even noticeable.
I don't see any conflicts here. As I said above, round-based game or any other "slow" game could take "fast" protocol.

"Slow" protocols are exactly "acceptable" for some games, just because they are easier for development.

Share this post


Link to post
Share on other sites

Faster systems aren't "the same, but faster". They usually have limits on what can be serialised and how much, to keep message sizes down. They may place requirements on your game to store several copies of state, to be able to rewind and replay from previous positions, or to be able to extrapolate from them. They may expect you to be sending a constant stream of redundant updates so that if some are dropped, it doesn't matter. None of that makes sense for a slow turn-based game.

Share this post


Link to post
Share on other sites

how to handle interpolation or extrapolation
I still don't see any difference in terms of protocol. It could support interpolation or extrapolation, so each game will decide if it's required for them or not and which exact setting they should use for timings and so on.

They usually have limits on what can be serialised and how much, to keep message sizes down.

But we have very complex fast games which have to send lot of data and they are working pretty good – so, it's possible. Of course, such protocol should not be used to, say, sending a media files, but we're talking not about some sort of The Ultimate Protocol, we're talking about pretty narrow use cases – multiplayer game client-server sync, basically.

Share this post


Link to post
Share on other sites
We can't take slow protocol for a fast-paced game, but we can easily take fast protocol for a slow game.

You do realize that a "slow" protocol is exactly as fast as a "fast" protocol? (Well, that's not true as it stands, but read on!)

 

A "fast" protocol is primarily not fast because it is any faster. This is, by the way, not limited to user protocols, it is a common misconception about TCP and UDP as well. TCP is exactly as fast as UDP. Except when reordering or packet loss happens.

 

A "fast" protocol is fast because in presence of packet loss it simply moves on and ignores the lost packet rather than stalling. That, and of course a "fast" protocol is paying a lot more attention to squeeze out the last bit only sending the absolute minimum necessary, leaving anything redundant out -- because only so and so much will fit into a packet.

 

That's OK for e.g. a shooter where there are not many interesting things other than "this single opponent moves to xyz" and "shoots at you", or "subtract 1 from life points" happening, but they happen in rapid succession, and woe if you don't get them in time.

 

But in e.g. a "typical" MMORPG you may as well have to transmit information like "This guy at XYZ is wearing a holiday gift hat, and the premium Soul Reaper armor with the Burning Sword of Z'kaaah. Right now he changed clothes and did the 'obscene gesture' emote. The other 45 guys on the screen are wearing...".

 

Not all of this information has the same importance (for example "orc hits you with axe" is certainly higher priority than "guy 20 meters away changes hats"), and usually there is some rather hefty prioritization going on. While surely able to "afford some more", still a MMO has to pay attention to bandwidth. Just do the math and see what happens when you send out two kilobytes per second to a thousand users all day long. Over a month, that's 5 terabytes (not counting network headers, traffic in the other direction, or other stuff).

Edited by samoth

Share this post


Link to post
Share on other sites

A "fast" protocol is fast because in presence of packet loss
Sure, but what does it change? If it's fit for fast complex games, it will fit for slow games anyway.

 

And, in addition, "fast" games still need to check states and so on "to be sure" - so, they still have to use some kind of "slow" protocol sometimes. And all this could be a part of common protocol, which has stuff for very fast messaging, for "reliable" messaging and how both are related to each other.

 

Again: I still don't see any conflicts why we can't take a "fast" protocol (I mean, the one which used for a fast complex game) and use it for slow/simple game.

Share this post


Link to post
Share on other sites
Sure, but what does it change? If it's fit for fast complex games, it will fit for slow games anyway.

Not necessarily, the design between using a "reliable, ordered" and "unreliable" service is an entirely different one.

 

And, in addition, "fast" games still need to check states and so on "to be sure" - so, they still have to use some kind of "slow" protocol sometimes. And all this could be a part of common protocol, which has stuff for very fast messaging, for "reliable" messaging and how both are related to each other.
Raknet has that. But of course, the ordered, reliable channels are "slow".

Share this post


Link to post
Share on other sites
Not necessarily, the design between using a "reliable, ordered" and "unreliable" service is an entirely different one.

Yeah, but we're talking about common protocol, not specific implementation. So, each specific game could decide if it's needed to use one or another part of the protocol or not.

 

 

Raknet has that

Raknet is specific implementation, not sure they have their protocol specs (please correct me if I'm wrong). Also, AFAIK, Raknet is not yet supported anymore.

Edited by norlin

Share this post


Link to post
Share on other sites
Yeah, but we're talking about common protocol, not specific implementation. So, each specific game could decide if it's needed to use one or another part of the protocol or not.

Once you start doing that, you don't have much of a high level protocol yet. If you strip back all the different high level options and just keep the common parts of the protocol, you end up with UDP and TCP, basically... :lol:

 

To go into details - some games may replicate object movement as:

* a series of position/orientation snapshots, combined with interpolation/extrapolation.

* a spline with start time / arrival time.

* a command of the movement destination, with the assumption that every client will deterministacally move the character to that destination in the same way.

* a mixture of the above.

Some games may do this as server-authoritative, others may be client-authoritative but broadcast by a server, others may be client authoritative but cryptographically checked by peers in a p2p network...

 

Often when designing online multiplayer gameplay, the realities of what you can and cannot synchronize across the internet (within your client min-spec connection, such as common DSL bandwidth figures, and within your server budget) often end up impacting gameplay decisions. You may want to implement "gameplay mechanic A", but your tech people tell you that it will use up 50% of our bandwidth budget, or increase server hosting costs by two-fold, etc... so you end up going with "gameplay mechanic B" instead.

Designing game rules isn't just about being a great game designer and knowing what players want to play, but also dealing with all the real world constraints, such as technological limitations.

 

 

...however, yes, you could write a networking library that implements dozens of different options. Games could then pick the algorithms they want, and mix-and-match different library features to fit their game. Writing that kind of library is a lot of work though! :)

Edited by Hodgman

Share this post


Link to post
Share on other sites

Ok, I've give up until I'll learn all that stuff more closely to find specific arguments. But I'm still sure it's possible to design a more-less high-level common protocol for a multiplayer games...

 

Anyway, thanks everyone for discussion and patience :D

Edited by norlin

Share this post


Link to post
Share on other sites

Raknet has that

Raknet is specific implementation, not sure they have their protocol specs (please correct me if I'm wrong). Also, AFAIK, Raknet is not yet supported anymore.


We already explained above why there is no requirement for a standard protocol, because interoperability is of near-zero interest.
 
The only significant benefit to a game developer from any sort of standard for networking is to be able to leverage existing code. Raknet is (one example of) that existing code. Enet is another. The systems built into Unity and Unreal are yet more. Then there are things like Photon (for Unity), etc. Off-the-shelf solutions exist, but they aren't standard, and never will be.
 

If it's fit for fast complex games, it will fit for slow games anyway.


A system where you have to work around or switch off a ton of unnecessary features is not a good fit. Edited by Kylotan

Share this post


Link to post
Share on other sites
I think your confusion comes from a flawed assumption.
You assume that, if there is a problem that needs solving more than once, a standard protocol will serve the community best.
(By community I include everyone involved, including those that want to make money making software.)

This is not always the case. Standard protocols are great when different pieces need to interoperate.
We have standards for bolts and nuts, so that manufacturer of part A can specify "use a M6 hex head carriage bolt of length 40 mm" and manufacturer of part B can slot that part in, without them necessarily knowing about each other ahead of time.
We have standards for web browsers, so that manufacturer of web server A can specify "I support HTTP version 2.0" and manufacturer of web browser B can connect to that server without knowing about A at all.

Open standards/protocols arise from a need to interoperate without bilateral agreements. They don't just arise from a need to solve a particular problem; that's a necessary but not sufficient precondition.
Compare to numerical library code. There exists no standard protocol or library interface for numerical code, even though there exist open source numerical libraries.
There are a number of libraries, like BLAS, and IMSL, and Intel MKL, and others. They don't use the same API, because there is no value to vendors to being able to swap one out for the other.

When it comes to games, there is zero pressure for the open standard, because it's simply not the case that two different manufacturers need to interact with each other without knowing about each other.
Understanding this distinction is crucial to making sure that you solve the right kinds of problems! It's one of those things that distinguish a very experienced software engineer from someone who can just write code.

Share this post


Link to post
Share on other sites

This topic is 405 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this