Sign in to follow this  
CamaroFloyd

Peer based MMORPG Network Engine

Recommended Posts

Dear Fellow Network Programmers, For a couple years now I have been researching ways to make monthly costs for playing a MMORPG decrease if not disappear. I've been looking into creating a P2P (Peer-to-peer) network engine that allocates server tasks that would originally be placed on the Central server, and place them on the clients. This would in turn act like a distributed networking scheme, but it still implies the infrastructure used in MMOG mainframe cluster networks. I will attach the basic scheme below, and if anyone who is interested in Network Programming, please read over it. Thank you: To start off, I must let you know that I have a very difficult time explaining complex things. So bear with me. First of all, each computer playing the game will have two different aspects called the Server and Client side. The Server side is stationary in terms of what it is doing in the game world. The client side however, is dynamic, moving about the world (hence the player inside of the game). So for the following explanation, whenever I mention the server, that is the server SIDE of a computer logged into the system. As such, when I mention client, that is the client SIDE. So, finally, the great explanation... Imagine a gigantic square (representing the world map). This large square will be handled by the game host (positional, character status, etc) and while each person joins into the game, they will handle a HALF of that square. When a third person joins, they will encompass (or serve) the area (or half) that is the most populated (and there is a ratio for this to be compared, called the PHR, or the Population Handling Ratio, where the server numbers for an area are compared to the clients in the area. So for right now, we have 2 computers on the network. One, the game host, and two the 1st player in the game. But this does not mean that the game host is not a player. So in actuality, there are 2 players and 2 servers in the game. The game host is handling the entire Map, but only calculating positional status for one side of the map, and counting the population or PHR for the other side. THE SECOND computer is handling ALL of the (for explanation's sake) right side of the entire map. His job is lessened though, because he does NOT have to handle ANY of the LEFT side which is what the 1st player (or the Game Host) is already taking care of. Now, when a third player joins, or any additional player at that joins, he will have to go down the hierarchy, first at the Game Host, and then the second player and decide which one is in the MOST need of a server. More than likely though the area where THAT player starts (or spawns) inside of, he will also be serving for. This is also where he begins (on the hierarchy) in order to find areas where he should serve. AND THEN he will move into other areas searching for a smaller PHR ratio (servers/clients). Once he finds an area where there is a need for more servers, his server side will take over that area. If you have not already picked it up, each computer is taking HALF of an area of someone elses area. So for instance the second player that joins takes over one of the TWO sides served by the Game Host (or the first player). This continues to happen further on as more people join into the network. So, if Game Host's Right side half is served by Player 2, and Player 2's Bottom Half is handled by Player 3, and Player 3's left side is handled by Player 4, and so on, this means that THIS AREA is the most populated and has the most servers (while at the same time, has the most clients). Now, when a client moves around, his computer is sending positional and character status through the network hierarchy to the computers linked above it. This also means that if the client is in the same area handled by his/her SERVER side, then the positional status does not need to go far through the hierarchy (hence, reducing network traffic among all the computers from within it). However, if that person ventures out of his server domain area, he risks increasing his latency in receiving information from servers around him in that area. To fix this however, there will be methods in place to reconnect the server side to different parts of the world, and the other servers will adjust accordingly to this reconnection. For the most part, people in one area (client side) will be seeing OTHER people in that same area (server side). But as long as these individuals remain close to eachother, their latency will be minimized. But again, with special background tasks, this problem can be avoided I will have further details to come. For right now, look over this and send back any additional comments Thank you again for the support -CamaroFloyd p.s. I have already created a thread on Gamedev.org : http://www.gamedev.org/yabbse/index.php?board=1;action=display;threadid=1423 However, no one on there is very knowledgeable about the subject. So I came here in hopes that I might find someone willing to help me put together this project of mine. If you're out there send me an email, or a response on this thread. Again, thank you.

Share this post


Link to post
Share on other sites
the biggest problems with P2P MMO systems:

bandwidth: most residential broadband connections are assymetric. i.e. they only offer a fraction of the bandwith to uploading that they do to downloading. thus you're servers will have very narrow outgoing pipes and thus very slow response times

reliability: in your scheme any given server is exclusively responsible for updating a specific section of the map. a hacked server could leak other player's money/gold to the player of that client. thus you need redundancy. any given area needs many clients all managing it and checking results with eachother. you also need the redundancy so that the game doesn't crash when one server's host dies (local network outage, machine crash, etc).

ability to make connections: most residential broadband access will be behind some kind of NAT or firewall. as such you cannot make incoming connections unless the user specifically has those ports open. this one should be pretty trivial to get around: don't allow user to play unless ports are open.

latency: latency is going to be horrible. servers have to check with each other over tiny pipes, then they have to relay the information back to the clients inhabiting their zones. that's 2X the communication that normal MMO servers have to do over a tiny fraction of the bandwidth.

I definitely think this is an interesting approach to hosting MMOs. the road is certainly frought with peril but is something i have toyed around with myself. the biggest problem, by far, is going to be latency. i mean look at the response times for simple text queries on common P2P apps. that's a minimal amount of data and the response times suck big time.

-me

Share this post


Link to post
Share on other sites
theres one other concern that jumped out at me. you say the goal of this is to reduce or completely remove the monthly fee that comes with an MMO. theres one problem though - this fee pays towards more then just server maintence / bandwith. your also paying game designers and programmers and modelers (and etc...) to continually add, change, and maintain content to the game. not to mention bug fixes and balance issues and stuff.

however, i guess a more realistic goal would be the "reduce cost", and not remove.

Share this post


Link to post
Share on other sites
Well, let me first start off by thanking you... you're the first person Ive come across that seems to know what they're talking about. Most people want to criticize this idea. But I believe that it has potential, even when it is faced with a lot of these down-sides. For example, when you discussed certain areas it could be flawed in, I tried to come up with a couple of arguments. Maybe Im wrong, but tell me what you think...

One of the down-sides was bandwidth, where most of the people who use the system (hopefully) will be ones who have some sort of middle-speed connection, be it a cable modem, T1, or DSL. This means that their outgoing communication will be limited to say... a few kilobytes (10-30k). Now, this is definately true, but with what Im talking about, the user would at most have 3 connections. One going to his super-user, and another two going to his "children". Now, I also discussed that by keeping the users client sides close to eachother (in the virtual world) they will decrease latency and at the same time, decrease outgoing and incoming communication. However, if they choose to roam around inside of the world map, they risk increasing that load, and increasing traffic to and from their computer. Now, this is certainly something to think about when making the system, but definately not impossible to accomplish.

Next was the connection establishments. However you already picked out a solution to that ;).

The third was reliability. This is also true as a problem. However, as I had mentioned on my Gamedev.org thread, you can have a back-up scheme that works in two ways. First, the user trying to send a message to a computer above it will notice that that computer is disconnected (or not responding). In this event he will have (already) an ip address stored for that connection to "hop" over the "missing link" and send the message to the computer connected ABOVE that terminated computer. That message will also include some sort of distress signal flag of some sort. AND finally, the second back-up plan that looks VERY similar to the classic MMOG server scheme. The plan is, simply, that if, say, 10 computers above a user suddenly disappeared from the network, then the user who is by itself will have THE GAME HOST's ip address. Now this means that the Game Host would have to stay dedicated to hosting the game. Now as much as this seems like the classic MMOG network environment, it would be the last resort for the user to go to. This (the back-up schemes) also adds complexity to the system, but if you are creating ANY system, you are bound to run into problems you will have to face and create solutions to (which in turn means that ANY system IS complex).

Now the last one was about latency, but again, you could answer or argue that with what I mentioned in the bandwidth problem. Other then that, if I missed something, let me know.

-CamaroFloyd

Share this post


Link to post
Share on other sites
Quote:
Original post by graveyard filla
however, i guess a more realistic goal would be the "reduce cost", and not remove.


True, but only if the game requires additional input after it was released. You can have smaller games running on this system, say, a 2d tile mmog that a small team of programmers have put together. But in theory, and in practicality, Ive been trying to, well, reduce the COSTS of maintaining or keeping up a MMORPG game. Because, as we all know, there are MANY mmorpg games that fail to keep them up because of the cost TO keep them up. I feel it is my duty, and responsibity as a programmer to find new and innovative solutions to these apparent problems.

-CamaroFloyd

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by CamaroFloyd
For a couple years now I have been researching ways to make


It's a fascinating discipline (dist systems) and worth researching. Unfortunately, most of what I have to say is crushingly negative, because you clearly are largely ignorant of the prior art in dist systems. If you find some books you think might be useful as background knowledge, post the ISBN's or titles (or just amazon links) and I'll give you a 2nd opinion on which are going to be really helpful and which are a waste of time.

Quote:

monthly costs for playing a MMORPG decrease if not disappear.


Most of the monthly costs are nothing to do with the network or server.

I'm distressed that you've spent years on this without having done the first step: finding out precisely what proportion of costs aer due to which causes. But hopefully you've learnt some other useful things along the way.

Quote:

However, no one on there is very knowledgeable about the subject. So I came here in hopes that I might find someone willing to help me put together this project of mine. If you're out there send me an email, or a response on this thread. Again, thank you.


Well, I am (specialist in distributed systems programming; I even hold patents on it). And I can assure you this causes more problems than it solves.

Your biggest ones are:
- security
- latency
- administration

You have no control whatsoever of the game ("you" as developer).Therefore, the game will most likely be complete crap, with everyone cheating, or massively incompatible versions. There are ways around this, but they all hugely increase latency (which is already a show-stopper).

Latecny will kill you time and time again: your system introduces spectacularly huge latency. Don't even dream of solving anything else until you've calculated the induced latency your scheme creates (it's huge; feel free to do the calcs and post here if it seems small and we can point out what you're missing).

More specifically, you have come up with a trivial algorithm (each newcomer gets half of someone else's responsibility) and completely ignored the one difficult part: coping with the massive fragmentation as people join and leave from millisecond to millisecond.

As soon as a player starts walking around, you'll lose all cache coherency, and that's going to make your latency problems 10 to 100 times worse.

Off the top of my head, I would guess your system (as described) would run at approximately 10 seconds per frame (i.e. 0.1 FPS) for a decent MMORPG with a few tens of thousands of players. You can hide a lot with smoothing and prediction - but not THAT much. I have actually known some commercial MMOG's (which got canned before release) which tried similarly naive (I'm tempted to say "stupid" since they spent millions on something they didn't even bother to research before hand) algorithms and 1 FPS max speed is not unusual.

c.f. DNS, which has always had major problems with achieving latecny on a massive peer-to-peer dist network. You should be wondering "if even something as trivial as DNS has problems with latency, do I have any chance at all?"

P2P networks are great for many things; they are totally useless when it comes to low-latency requirements - hence mostly useless for games with more than a few players (and a foolish idea even then).

redmilamber

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by CamaroFloyd
Because, as we all know, there are MANY mmorpg games that fail to keep them up because of the cost TO keep them up. I feel it is my duty, and responsibity as a programmer to find new and innovative solutions to these apparent problems.

-CamaroFloyd


I'll give you some help here.

Cost of one programmer for 12 months: $50k
Cost of one programmer if it's a self-owned indie and the programmers are all co-founders: $15k (living costs, basically)

Cost of server hosting + bandwidth for an MMOG for approx 10,000 players for 12 months: $5k

And that's ignoring all the other costs. Look, servers are so chepa they're practically freee these days, whilst bandwidth is damn cheap if you:

a. Buy in bulk (buy by the terabyte, or preferably by the tens of terabytes)
b. Shop around, negotiate a good deal, pay 12 months in advance

Oh, and I've made the programmer costs lower than they really are and the server costs higher than they really are. In practice, it's often a larger difference.

PS: please note: most MMOG's that claim their costs were too great ARE LYING! It's too embarassing to say "we're too damned crap to make the code work" or "our game idea was crap, and even though everyone else makes lots of money easily out of crap MMOGs, we failed" or "this costs a lot more than we realised because we were too lazy to do a proper budget before we started".

So, instead, they say "it's just too expensive to run" because that makes it sound less like they were stupid. Or, like MS, they say "the game we were writing was excellent and amazing but we decided that by the time we released there would alreadt be several other amazing games too and we didn't want to reduce our profits by competing with them" - as if:

a. there was *nothing wrong* with their game
b. they don't normally release products if there's any competition

It's just BS. If you listen carefully you can hear a "swishing" noise - that's the sound of the games company covering their own ass.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by hplus0603
Go to gamasutra.com and see if the "peer-to-peer cyberspace" articles by Crosbie Fitch are still online. They were written sometime around 2000, I think.


Yup, Crosbie is mental. People have been pointing out the obvious problems for years but he won't accept them. Although he has recently started adapting and coming up with slightly less insane and stupid ideas, which is nice. Maybe eventually he'll come up with something feasible. Or find a way to live forever and live long enough that technology makes his otherwise impossible ideas actually work (just like tech eventually made VB run fast enough to do most apps at a decent speed)

Share this post


Link to post
Share on other sites
Dear All Readers,

It is sad, really, to see the people that criticize and completely turn down the availability of another solution to an important issue. As much as I have tried to point out what I am trying to do, I recieve an increasing amount of negative garbage trying to shoot down the possibility of something innovative. One user whom responded to my thread has done this openly. How much this user knows about shaping the future of computing, or changing how current technology is carried out, is unknown to myself. However, I would let you know, I have a very open mind when it comes to things I love. And you can laugh at that as much as you want. But a true programmer does not mock other programmers simply to cover up his own mistakes or build himself up. This is NOT about ourselves, and it never was. What we have forgot is that we have taken on the responsibility as programmers to forgo an exploration of the unknown, or the imposssible. Great leaders in the past such as Galileo did NOT criticize or dispute FUTURE ideas, he explored them and shaped them into beliefs and understandings that we hold today. Now, you call me a developer of "...complete crap" and you call me naive, but what you DON'T do is understand my motives. If I am so wrong, show me a better way than shooting down a man's hope for the future. Again, you can laugh at this, but what, if anything, do we have in this world except hope?

I have delved into this project believing that there are ways unknown to us that are still unexplored, and I drive myself to find these things. And I hope that you would see why, and help me instead of tearing apart someone's ideas. Now, I also understand you posted these remarks anonymously, hiding your identity to allow yourself to express how you feel (bluntly) but this does not help ANY situation. It's only an attempt at bringing someone down to believing there is nothing to have faith in. Please, I ask, that not only those individuals I speak of, but anyone who feels the same as me ( be it programming or anything ), when you feel for something or you have a passion for something, don't let it go, not even when faced against those people who anonymously try to tear you a part. If there ARE people out there that try to tear you down, I wish that you know better than to act indescent and corrupt like they do, and see that there are people like myself that want to see things move forward and into the future.

I hope that what I have said would have hopefully put a smile on your face, because it did for me, and I thank this site for allowing me to do that. Thank you. Everyone, please, have a good one, and take care.

-CamaroFloyd

Share this post


Link to post
Share on other sites
Either I fundamentally misunderstand what you're trying to do or I see some serious flaws.

A limit of two to three clients per server is impractical in a variety of instances. Consider a major town center where upwards of 100 players can be in a geographically limited area at the same time. In this case, if each server is only supposed to serve two to three clients, you'd need to have 30-50 servers handling that small area. The problem is, each of the 100 players needs to know the positional information of up to 99 other players. In this case, you'd need to have the servers intercommunicate with each other which is problematic by my understanding: most importantly, this radically increases the amount of bandwidth and latency used to propagate player positions - instead of PlayerA->CentralServer->PlayerB + PlayerC + PlayerD... it's PlayerA->SomeServer->OtherServer->PlayerB, PlayerA->SomeServer->AnotherServer->PlayerC,
PlayerA->SomeServer->YetAnotherServer->PlayerD.

In an abstract sense what you're proposing is akin to a very complex equation that simplifies down to a known, simple equation. The p2p method you suggest is a complex way of handling the problem that should "simplify" down to produce the same results as the client-server model. The difference is in the inputs (bandwidth, latency, monthly fee, development time, etc.). In this case it would appear that you're proposing a solution that greatly increases the bandwidth, latency, and development time necessary to solve the problem and leaves the monthly fee at about the same level. The point is, the model that's being used presently is about as good as anyone has come up with thus far. Certainly, pursue your goal, but not to the extent that some sound skepticism doesn't give you pause.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
I was anon because GAMEDEV.NET HAS CRAP COOKIE HANDLING / LOGIN HANLDING THAT IS BADLY BROKEN. Sorry to shout, but people keep throwing abuse (not you) or complaints (you) at those of us who have NO CHOICE but to be AP most of the time, and it's getting irritating. We don't do it on puprose: it's the admins who wrote a site that can't do what all the other community sites on the net have been getting 100% right for decades that are causing the problem!

Also, I actually signed my name on the post, so ... go figure.

Quote:
Original post by certaintragedy

In an abstract sense what you're proposing is akin to ... you're proposing a solution that greatly increases the bandwidth, latency, and development time necessary to solve the problem and leaves the monthly fee at about the same level. The point is, the model that's being used presently is about as good as anyone has come up with thus far. Certainly, pursue your goal, but not to the extent that some sound skepticism doesn't give you pause.


Exactly. If you can't handle the fact that most ideas are hopeless or pointless then you won't get far with research; a part of research is about having 100 crazy ideas and then knowing which ones actually have some promise, and following those. All I'm doing is pointing out the fundamental reasons why what you suggest is useless in the real world of today; in 2104 with completely different society, technology, and telecomms links your ideas may be not only feasible but superior to what is used today. But most P2P advocates don't ever bother to check what the telecomms links of today are actually like (most of them never bother even to learn the basic practicalities of telecomms, the standards and protocols, the well-researched problems and solutions) and so sit around dreaming up ideas that are useless in real life. Shrug.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this