Jump to content
  • Advertisement


  • Content Count

  • Joined

  • Last visited

Community Reputation

105 Neutral

About drainedman

  • Rank

Personal Information

  • Interests
  1. Continued from here.... Turns out I could actually get the $50k and built my own cluster across multiple machines with a total of 128 cores. The development has continued since and I have to say the results are fairly pleasing. The scaling was fairly successful and it will probably distribute out quite well. My approach was very similar to sharded databases, and although I rolled my own I suspect I could have used something off the shelf I would know it in enough detail. https://en.wikipedia.org/wiki/Shard_(database_architecture) e.g. MongoDB. I would do something like have shards that hold limited information of a geographical area (e.g. positions of player). Only certain regions would be replicated (e.g. neighbouring regions). A simulation client that runs external to the database but on the same local machine would do the simulation and update the shards. In any case I have rolled my own, and I think in terms of distribution its pretty good, of course the amount of interactions between entities will always be limited so I question whether its possible to use this for a massive FPS or anything like that but otherwise should support lots of entities.
  2. drainedman

    SpatialOS single shard MMO

    That's true. How might one attempt to build scaleable units with docker containers?
  3. drainedman

    SpatialOS single shard MMO

    For what its worth this seems to be based off Docker containers https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/. Which is interesting of itself and can get a bit more detail of the technical aspects. I don't make any claims to fully understand all of this, however my initial impressions are that this is more suitable to stateless load balancing (of web apps and so on) and rollouts of installations - rather than being an MMO platform and directly addressing the MMO issues. I cannot see any guarantees about state synchronization, speeds, etc. I tend to prefer to deal in hard numbers and not cuddly flow charts.
  4. drainedman

    SpatialOS single shard MMO

    I have always generally preferred to roll my own solutions where there is not a proven technology available. I think the reason behind this is that I like to fully understand how my system works. I also like to deal in precise quantities, I like to know what is and isn't possible. Using 3rd party stuff I can never be sure if I'm using it wrong or if the "technology" is limited. For example Erlang gets recommended a lot but I just don't see the speed advantage of using Erlang for low latency distributed simulation for instance. Perhaps I just don't like Erlang. Maybe the supposed advantage is that its easier to use but I'm already invested in my own pet technology so that its easier doesn't really hold true for me either. In any case my approach is not treading new ground, which gives me some optimism. I have read papers on similar stuff done years ago.
  5. drainedman

    SpatialOS single shard MMO

    Indeed MPI is just a tool and not a complete solution. SpatialOS is a "solution" platform but it remains to be seen that it can do the more demanding applications such as a huge Unreal tournament, for example. Of course as you pointed out we have seen this kind of tech many times before. Pikkoserver, Shinra spring to mind as the latest failures. I don't think its an impossible feat as such, just a bit limiting with current hardware. I don't really know! I suspect the answer lies in a more hardware solution than software however.
  6. drainedman

    SpatialOS single shard MMO

    This is how my tiny brain understands the problem. Ultimately locality is at the core of the it (server side). Suppose an entity is processed 30 times in 1 second. In order for this entity to process its behaviour within that slice of time it must take in information of its nearby space. For example a brick falling through space will need to know about its neighbours so it can go bumpity-bump-bump with other bricks. We can cheat a bit by generally disregarding entities a long way away to reduce the number of interactions from squared to linear. When working within a single process we can do entity neighbour lookups very quickly as RAM access is pretty quick (we have seen this used to great effect with CUDA physx demos and so on). 10,000 entities will mean 300,000 neighbour queries on top of physics, behaviour calculations etc per second. Quite manageable within one process. However, to scale up to more entities we want to split the workload across two or more nodes and the processes cannot access each other's RAM. non-local entities (entities from different process) must talk to each other by some other medium. Quick, large distributed shared memory isn't an option with current hardware (although surely some hardware guru could build it), so we use something like standard networking. Because of this our communication speed between non-local entities has dropped by a factor of about 200 or worse. To compound this latency we find that highly dynamic environments such as MMO's will vary the load which can mean high volumes of traffic in concentrated areas. Some entities will travel insanely fast across multiple nodes (speeding bullets, airplanes, cars). Also some very selfish entities are particularly inconsiderate and want to exchange HIGH volumes of data and want to do it instantly. Transactions in market places springs to mind. Or perhaps a car with a 10 hour pre-planned route. Or an entity with a daily schedule. We can't really ever get around these limitations until we (somehow) increase the speed of access across all nodes to be as quick as if they were all a single unified node. Meanwhile designing the game/simulation is critical to making the whole experience balance out. I don't even think its possible in the generic sense to make a scaleable MMO - with current hardware. Actors, services, workers, etc I view as a kind of syntax flim-flam, froo-froo. It does little to address the limitations. My personal pet favourite tech to tackle this problem is MPI https://en.wikipedia.org/wiki/Message_Passing_Interface
  7. drainedman

    SpatialOS single shard MMO

    I can't afford $50k but I am planning on building a beowulf cluster to support a kind of MMO with 100k+ entities (not all human but all are persistent). I'm just going to go for a simple method - each 2 km sq region is handled one process. Every process sends sync messages to its neighbours via network messaging. Only edges of the regions are kept in sync. I'd like to run this over a VLAN to separate network channels. Of course this has lots of limitations but if I get this far then I will implement some load balancing e.g. split the regions with heavy loads into 1km sq regions, etc. So long as not too many objects converge in spot and do too many interactions it might be ok.
  8. drainedman

    SpatialOS single shard MMO

    What if you had just one server with 128 cores and a huge shared memory? Surely that would support hundreds of thousands of players.
  9. drainedman

    SpatialOS single shard MMO

    Ah pretty disappointing if it can't do low latency. They kind of sold it as that though.
  10. Has anyone tried SpatialOS? https://improbable.io/games I'd like a different perspective I don't pick up a lot just by thinking about it myself. Is this stuff quite similar to Pikkoserver though? Are workers/services really the way forward. I don't see any detailed explanation or case studies, just presentations.
  11. drainedman


    As in BigWorld server...how well does it work? It can supposedly support single shards with large player counts.
  12. drainedman


    Anyone have any experience with BigWorld?  
  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!