On 20/06/2017 at 7:31 AM, drainedman said:

I can't afford $50k but I am planning on building a beowulf cluster to support a kind of MMO with 100k+ entities (not all human but all are persistent).

I'm just going to go for a simple method - each 2 km sq region is handled one process. Every process sends sync messages to its neighbours via network messaging. Only edges of the regions are kept in sync. I'd like to run this over a VLAN to separate network channels.

Of course this has lots of limitations but if I get this far then I will implement some load balancing e.g. split the regions with heavy loads into 1km sq regions, etc.

So long as not too many objects converge in spot and do too many interactions it might be ok.

I am currently building a mini cluster of raspberry pi's. So far I got two of them but plan to expand to four.

Each raspberry pi has 4 cores and 1 GB RAM and 16 GB SSD drive.

You could easly support 20,000 - 40,000+ concurrent players on that, but it heavily depends on the game design.
When doing single shard multiplayer games, it is critical to avoid choke points, such as having one capitol city or one location for the auction house where players tend to group together.

I made my own distributed server in Erlang and on an 8 core desktop computer I could support congestion of about 2 groups with 2000 players at the very same spot. This would be similar to players forming a warband in WoW and running around killing stuff.
With those 2 groups of 2000 players each, the same server was under heavy load but still responsive, I tried to fire up a third group of that size but then things started to lag badly.

In an other test, I spread the players evenly about the world, I had divided the world (1024x1024m) into smaller chunks of 64x64 meters. With about 60 players in each such chunk the server were running smoothly handling around 12,000 concurrent players with latencies between 10 - 50 ms.

I have not yet tested the performance on my raspberry pi cluster, but I am working on it now and should have some interesting results to share later this  summer.

Supporting 100k+ entities were most of them are static or mostly inactive should not be very hard, but it is time consuming to build such software unless you use something already existing.

You could easly support 20,000 - 40,000+ concurrent players on that

While the Raspberry Pi is amazing, it's not THAT amazing. For example, the Ethernet is a 100 Mbps interface sitting on an internal USB hub. And calling the MicroSD card a "SSD" is ... a little optimistic :-)

I bet, if players aren't doing much, and you don't have real-time simulation requirements, and bandwidth is carefully managed, you could do thousands per Pi. That's still pretty amazing!

For example Erlang gets recommended a lot but I just don't see the speed advantage of using Erlang for low latency distributed simulation for instance.

Erlang gives you the advantages of immutable data. It also gives you the advantages of ultra-cheap "processes" (more like "fibers" but the immutable data means you can't accidentally someone else.) It also gives you the advantages of being able to upgrade the running code in-place. No rolling re-starts, no socket close and reconnects, just keep on going with the new version of the code. (The immutable-isolated-data concept makes writing these in-RAM migrations possible.) Finally, it gives you very small-size, threaded garbage collection, because each little micro-process has its own heap that's collected in isolation.

The upgrade-in-place is actually quite hard to do with DLL re-loads in C/C++, or most of any other language. The other bits are possible in other languages, with slightly different trade-offs. If you're interested in systems, in general, and haven't used most of those features (especially immutable-data functional programming) on previous projects, it's at least worth checking out for learning. It is a very different paradigm, though (if you haven't already used ML/Haskell/OCaml/F#) so expect going to be very hard and uphill initially. It really takes time and effort to learn, like anything else that is actually big, different, and worthwhile.

That doesn't mean Erlang is the right choice for you. The IPC is low-latency, but cross-node communication uses TCP. The VM generates native code for execution, but the constant overhead is noticeable (Erlang is slightly slower than Java in my experience.) Not every project needs 100% uptime even through deploys and rollbacks. But, it represents a fundamentally different approach to the problems faced by distributed systems developers, and thus, it's quite worth learning in detail.

9 hours ago, hplus0603 said:

 

 

While the Raspberry Pi is amazing, it's not THAT amazing. For example, the Ethernet is a 100 Mbps interface sitting on an internal USB hub. And calling the MicroSD card a "SSD" is ... a little optimistic :-)

I bet, if players aren't doing much, and you don't have real-time simulation requirements, and bandwidth is carefully managed, you could do thousands per Pi. That's still pretty amazing!

 

I was referring to my Raspberry cluster of 4 Raspberry's, maybe that was not clear in my post, sorry about that. But yeah, the numbers heavily depend on the type of game play you have in your game.

2 hours ago, drainedman said:

For what its worth this seems to be based off Docker containers https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/. Which is interesting of itself and can get a bit more detail of the technical aspects.

Docker is largely a cheaper way of running "virtual machines" of software in isolation from other software. It makes building the "machine images" (containers) straightforward, and has a number of orchestration methods to put containers on physical hardware. The draw-back, compared to VMWare or Xen or whatever, is that all the containers have to run on the same kernel, and that kernel needs to support containers in turn.

You can run web servers in containers. You can run game servers in containers. You can run memory caches in containers. You can run proxies in containers. You can even run desktop software in containers (if you forward the appropriate desktop.) You can also run databases in containers, although the I/O virtualization ends up making that so slow that nobody really does that for real.

The fact that something "uses containers" or even "is built on containers" doesn't really tell us much at all about what it does, other than that it presumably has some way of breaking up work into smaller, hopefully scaleable, units.

The other thing containers do, is deliver a "complete working piece of software" in a single download.

All the libraries and data files and whatnot needed by the software in the container, is typically included inside the container image itself.

Thus, containers are also a convenient way of delivering pre-configured software for various tasks, even if those tasks aren't inherently scalable. Instead, what you get then is the convenience of "download a thing, click run, and it will (hopefully) Just Work (tm)"

Thus, "it uses containers" tells us pretty much nothing about the innards of the system, but it does tell us that it's probably easy to download something and start it up, and it's probably easy to start up many copies of it. Whether those copies will efficiently form some kind of larger cluster, is totally up to the implementation of the software on the inside.