On the note of "web-app stuff applied to an MMO server", has anyone checked out NetEase/Pomelo (an MMO engine built on top of Node.js)? For what I've seen it has a pretty straight forward way of encapsulating distributed servers (although it seems to still suffer if too many players congregate in one area). Sadly, I haven't been able to find any real analysis apart from these slides: http://www.slideshare.net/xieccy/pomelo-jsconfasia So I don't really know how it handles server interactio and how would a physics simulation would run in there.

offtopic:

I bet through some clever trickery and using player locality you could engineer a system of virtual boxes or spheres of players that overlapped and allowed you to grow or shrink on demand.

Awesome, I recall thinking about something like that when I "dream of making an MMO some day" in my teen years. I'll have to try it out when the day finally comes.

I'll reiterate my earlier cynicism: in the realtime massive distributed space, until you have millions of players running on Solution X, you can't believe any claims about whether or not Solution X is actually feasible.

Even the leading experts in this stuff have a hard time predicting how well something will scale. If it hasn't been shown to successfully host a nontrivial simulation with a very large number of concurrent players, it may as well not exist.

I could say this is very true (it is not even cynism but calm understanding) and it spans over whole programming, prepare for some words of wisdom:

though programming seem to be very logical activity, very big amounts of factors are always outside of your logical enfolding, in other words there are many factors which impacts on the outcome of your work but you do not know how they work * (often you even dont know what they are) - you could controll the known ones but only accept the presence of unknown ones - your more working test is experience here, as experience takes into account both the factors (those logically controllable and those unknown), (also extrapolation in thought (not thought, but extrapolation of it) rarely works though it is common human mind error)

* there are of course domains more predictable and those less predictable, ** mainly the more predictable ones are things that you was already doing (are deep into), not much predictable are the things you are yet not doing - but in general imo the majority of things people do are really strongly unpredictable/unexpected

** may seem that those more logical domains as hard algorithms are more predictable than those with not so logically controllable as a team work (why it is working gooda and why not etc) but in reality i think even those hard algorithms can be unexpected at it is always some way of looking on it from unexpected side that it will change your view again

(so this was small thoughts/essay on (un)predictability ;o )

As you said, character/character interaction is N-squared; connections (and web architecture) is all built around scaling out the N problem. No real-time physics simulation engine exists that scales out across machines along the axis of the number of cross-interacting entities, although they exist (expensively, see DIS) for making each separate actor extremely complex.

If your needs match those of Farmville, an EC2 based web solution is great.

Right, but I think you're suggesting that games fall into either "real-time physics simulation" or "Farmville", and my MMO experience is definitely more in the middle. For the most part, the only physics being simulated is a position and velocity for each character, and there can be significant lag or even jitter in those cases without any downside to gameplay as there are no aiming mechanics or similar. With that holding true, and movement being fairly easy to progressively degrade (eg. just by sending less frequent updates), the performance constraints are significantly reduced.

The main problem I would then face, with a system that doesn't need strong guarantees on physics, is that I don't see any easy way of decomposing functionality across multiple servers without the usual proxy or ghost entities to stand in for multi-entity actions. Changing every action into a 3 or 4 stage state machine working with async messages is unwieldy, especially if you need to lock each actor while a state machine is in an intermediate state. And implementing some sort of attack bonus that is conditional on the current state of an actor on another process... possibly not practical at all.

For the situation you're describing there Kylotan, you may be better off with a distributed STM approach rather than thinking of it as "purely async". They're not mutally exclusive.

http://avout.io/ is an example of something I'm using in development right now.

It's very similar to MPI approaches for cluster computing.