Jump to content

  • Log In with Google      Sign In   
  • Create Account

Kylotan

Member Since 08 Mar 2000
Offline Last Active Sep 09 2014 06:43 AM

Posts I've Made

In Topic: MMOs and modern scaling techniques

16 June 2014 - 02:35 PM


As you said, character/character interaction is N-squared; connections (and web architecture) is all built around scaling out the N problem. No real-time physics simulation engine exists that scales out across machines along the axis of the number of cross-interacting entities, although they exist (expensively, see DIS) for making each separate actor extremely complex.

If your needs match those of Farmville, an EC2 based web solution is great.

 

Right, but I think you're suggesting that games fall into either "real-time physics simulation" or "Farmville", and my MMO experience is definitely more in the middle. For the most part, the only physics being simulated is a position and velocity for each character, and there can be significant lag or even jitter in those cases without any downside to gameplay as there are no aiming mechanics or similar. With that holding true, and movement being fairly easy to progressively degrade (eg. just by sending less frequent updates), the performance constraints are significantly reduced.

 

The main problem I would then face, with a system that doesn't need strong guarantees on physics, is that I don't see any easy way of decomposing functionality across multiple servers without the usual proxy or ghost entities to stand in for multi-entity actions. Changing every action into a 3 or 4 stage state machine working with async messages is unwieldy, especially if you need to lock each actor while a state machine is in an intermediate state. And implementing some sort of attack bonus that is conditional on the current state of an actor on another process... possibly not practical at all.


In Topic: MMOs and modern scaling techniques

12 June 2014 - 04:33 PM

I wouldn't put so much trust on "how web developers approach scalability".


I appreciate that a lot of what used to be considered the state-of-the-art is now not considered best practice. But still, there are sites today deploying technology that services many more concurrent clients than single-shard MMOs can manage. The question is whether it would be possible for MMOs to do the same... or not. From what people are saying, the answer appears to be "Yes, of course... if you can overcome the complexity... which nobody is going to talk about in any detail". ;)

 

But still the problem remains that we have one socket per TCP connection and that sucks hard.

That's not really the problem I am trying to discuss though. Firstly, because you can distribute the front end servers quite easily. And secondly, because you don't necessarily need to use TCP for your main client connection anyway.

My experience of MMOs is that once you've done your optimisation on the I/O level - eg. getting your buffers the right size, perhaps using a proxy so that your game server is not spending half its time servicing network interrupts, etc - you'll hit a CPU cap with the gameplay before you hit a cap imposed by networking delays between the server and the player clients. Character interactions are O(N2) whereas your number of connections is only O(N), and the value of N is greater for character interactions because it includes NPCs. The cap in my experience seems to be around 500-1500 players per process, depending on how complex the computations for each player are.

 

Sure, at a very high level with distributed servers like Amazon EC2, these paradigms work. But beware that a user waiting 5 second for the search results of their long-lost friend on Facebook is acceptable(*). A game with a 5 second lag for casting a spell is not.


Sure. That's the backbone of my suspicion. The argument I had which inspired me to start this thread included the other guy saying that my traditional approach is quite obviously not widely used because it would cost half a million dollars per month on Amazon EC2. Trying to tell him that most MMOs - in the original meaning of the word - do not and will not run on EC2, would probably have been futile.


In Topic: MMOs and modern scaling techniques

12 June 2014 - 04:17 PM

If you have GDC Vault access, look up a talk by Pat Wyatt from GDC 2013 I think it was... maybe 2012.
 
If you don't have GDC Vault access, what's wrong with you?! :-P


It's far too expensive for me, unfortunately. sad.png

In Topic: MMOs and modern scaling techniques

12 June 2014 - 04:15 PM

First: Try writing a robust real-time physics engine that can support even a small number like 200 players with vehicles on top of the web architecture that @snacktime and @VFe describe. I don't think that's the right tool for the job.


I'd be inclined to agree. smile.png I was going to add the caveat of no real-time Newtonian physics, but I left that out to keep things simple. Perhaps that was a mistake.

You'd think that, given how much has been said on the matter, that there would be at least one instance of people talking about using different methods, but I've not seen one.


Personally, I've actually talked a lot about this in this very forum for the last ten-fifteen years. For reference, the first one I worked on was There.com, which was a full-on single-instance physically-based virtual world. It supported full client- and server-side physics rewind; a procedural-and-customized plane the size of Earth; fully customizable/composable avatars with user-generated content commerce; voice chat in world; vehicles ridable by multiple players, and a lot of other things; timeframe about 2001. The second one (where I still work) is IMVU.com where we eschew physics in the current "room based" experience because it's so messy. IMVU.com is written almost entirely on top of web architecture for all the transactional stuff, and on top of a custom low-latency ephemeral message queue (written in Erlang) for the real-time stuff. Most of that's sporadically documented in the engineering blog: http://engineering.imvu.com/


I have seen you post a lot about the There.com architecture and in general the games I've worked on have shared many of its characteristics, so that side is known to me.

However I don't recall seeing much detail on the IMVU stuff. I've started taking a look at that link (thanks) but I've skimmed through the whole thing and I can't see anything specific to the handling of state changes, or multi-node transactions, or decomposing complex behaviour into messages, etc.

In Topic: MMOs and modern scaling techniques

12 June 2014 - 04:05 PM

So this is an interesting topic actually.  The trend is to move reliability back up to the layer that defined the need in the first place, instead of relying on a subsystem to provide it.  

 

Just because the network layer guarantees the packets arrive doesn't mean they get delivered to the business logic correctly, or processed correctly.   If you think 'reliable' udp or tpc makes your system reliable, you are lying to yourself.

 

http://www.infoq.com/articles/no-reliable-messaging

http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.txt

http://doc.akka.io/docs/akka/2.3.3/general/message-delivery-reliability.html

 

 

Ok, but I'm still not following. Of course reliable transport doesn't mean correct business logic. But these are 2 separate issues.

 

Your first link basically makes the argument for application-level sequence numbers because it wants to preserve business logic even when the transport is down. That's reasonable, but it's not a key concern for most games. If game server 1 has lost its connection to game server 2 for some reason, you probably have bigger problems than making sure Jimmy's Magic Missile is propagating across servers properly. There's no guarantee you can recover in a meaningful way so it is arguably best to abort entirely. The exception would be for any sort of real-money transaction or similar, but as I hope was made clear in earlier posts, I'm happy with those being done in a more complex and yet more robust way, as they are a small minority of what a game server normally handles.

 

The other two links elaborate on this and say things like "The only meaningful way for a sender to know whether an interaction was successful is by receiving a business-level acknowledgement". That's fine, but not necessarily relevant. As above, a reliable transport layer will give you (short of astronomically unlikely occurrences) a guarantee one of two things happened:

  1. The message arrived at the intended place in an undamaged form. All is well if you coded the receiving method properly.
  2. The message didn't arrive, and your game is therefore broken.

With this in mind, the reliability guarantee given by the OS is going to be sufficient for any typical gameplay functionality.

 

So the problem with a pure message-passing approach remains that of whether it is practical to code all gameplay features to work in that way, given that it's not necessary.


PARTNERS