Jump to content

  • Log In with Google      Sign In   
  • Create Account


Member Since 23 Jul 2012
Offline Last Active Oct 18 2013 10:49 AM

Posts I've Made

In Topic: A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceshi...

06 March 2013 - 03:11 AM

In terms of spatial querying, you are competing against technology that most game-devs already have. In terms of parallel processing, many game devs already have solutions, but yes, there's still many trying to transition, who may be potential customers.


I don't think so. The whole approach, as outlined in the blog post, is that scalability comes from making parallelization and querying one and the same. If you say you have one or the other, or even both separately -- that's nothing to do with what we provide. We say that IF you have real-time db queries AND the db uses knowledge about the domain and the queries in order to parallelize your business logic THEN you get significant scaling. We're not trying to solve the problem of querying spatial data. We're solving the problem of scaling by utilizing the database. 


The main point of our approach is this: we're not utilizing parallelism to speed-up database queries. We're utilizing the database to parallelize (and speed up) your business logic



You're being silly and ignorant again.


Yeah, I'm probably being silly, but I don't think I'm ignorant. smile.png After spending a lot of times talking to potential customers in various markets, some patterns emerge.



I can tell you straight-up that console developers will be a very hard sell, simply because Java doesn't exist there!


All our middleware is server-side only. That's where you find the scaling issues we're trying to solve.


However, the number of large-scale MMO developers in the world is very low, so the chance of netting customers is also pretty low.


True. That's another reason why we're not trying harder to convince game devs to buy our stuff. For indie developers our software is free anyhow (depending on the game world size).

In Topic: A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceshi...

05 March 2013 - 11:41 AM

Better technology only makes a difference where the user experience changes significantly as a result, and the majority of the paying customers in the MMO world seem to be asking to play with increasingly smaller and smaller groups of people in order to better control their play experience. Thus the lure of the seamless world has lost its lustre somewhat. It's easy to mistake games as being a symptom or a demonstration of technology, but really the tech is the tool here, not the product.


I absolutely agree, yet expectation or demand is often a function of what's available. I think it was Henry Ford who said, "if I were to ask my customers what they wanted they'd say they want a faster horse". I don't think there was demand for video games before there were any, or for 3D shooters before Wolfenstein. New technology just opens new horizons for the more adventurous, imaginative early adopters. Then it needs a killer app for the late adopters to realize its potential.


I don't think EVE's success, or its amount, is simply a function of the architecture. I'm sure that if someone came up with a great game that employs a seamless world, a lot of others will follow. Still, that game would have to be great for many reasons; obviously just having a seamless world won't be enough.

In Topic: A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceshi...

05 March 2013 - 08:36 AM

The very phrase "may perform well enough for MMOs" gives an insight into how poorly the system was scaling before and how low their expectations were! The horizontal case for casual games is already well-provided for by other tech anyway, which is probably why Nphos doesn't seem to be around any more.


Yes, that's referring to Nphos's simple solution.As for Darkstar -- there was a prototype, but research was far from finished. At no point in the project, throughout its various incarnations, have they released even an alpha version of multinode. They believed they would be able to achieve terrific scaling, but all efforts on multinode were halted once Oracle defunded the project.


there would be problems using this tech for MMOs, as I assumed that was why you were posting here.


Yes, there would be problems, but also some great benefits to be gained. Nevertheless, we've given up all efforts to persuade game developers of anything. Turing and von Neumann themselves could not have persuaded a game developer. But the lucky developer who will try our stuff will get some great results, and we'll be all the more happy for him. That's not our business plan, though. If you give it a try you'll be happy that you did; if not -- well, I don't try everything, either.


ease of use for the programmers is probably more important in terms of getting the project finished and working.


Yes, but ease of use is different from sustaining habits. You're talking about habits, not ease, and, like we both said, game developers are conservative :) Many "web developers that don't know what a TLB is" were able to quickly wrap their heads around Clojure, Erlang, Go and Node and mostly reaped the benefits of better productivity, even at the price of changing their habits. But, to each his own.


Ultimately players are far more tolerant of shards that are half the size than single worlds with bugs.


Yes, because they only know what they're getting. I'd take that further. Players are more tolerant of games that haven't progressed much in terms of features and abilities in something like a decade; this particularly applies to MMOs. Google and Facebook are able to process in realtime several orders of magnitude more than they were a decade ago. MMOs operate at pretty much the same scale. 


If you'd care to hear my thesis about it, here it is, and it's completely different from yours, about risk and investments. The reason for conservatism is that AAA games, especially MMOs, are prohibitively expensive to produce, and the software development is mere fractions of the cost. It almost all goes to content. For this reason, incumbents like Blizzard and CCP have little to fear from small startups with a better software architecture threatening their position. A startup could build an MMO architecture far superior to Blizzard's, but it doesn't matter because they won't have the resources to produce as much content and compete with WoW or EVE. There are very few incumbents, all of them co-existing happily together. And that is why the big studios have little reason to constantly innovate when it comes to architecture. To push MMO studios to innovate you'd have to come up with a game that raises player expectations, and that would only happen if players like it, which, in turn, will only happen if the content is rich. So it won't come from a startup. Without this expectation from their customers, innovation happens at a glacial pace. No one is in a hurry. 


This is completely different in industries where you don't have such high content production costs, and so even the incumbents are constantly fearful of newcomers; so they must innovate. Engineers there are constantly looking for new techniques and approaches that would give them an edge over the competition. 

In Topic: A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceshi...

05 March 2013 - 04:59 AM

The actual code that's run on the GPU depends on both the shader bytecode and the command stream generated from the API.


Cool. I actually did not know that.


I also think there is a lineage difference. Web developers come out of HTML design and high-level application frameworks. They have never had a multi-year project die because of occasional one-second lag spikes. They generally wouldn't be able to explain how a L1 memory cache is different from an OS-level block buffer cache. They want to use whatever new shiny toy gets their site to look and work better for their test user -- and most of them don't really have that many concurrent users.


I'm afraid this is what many game developers believe but it is patently false. True, most web companies (one company one vote) match your perception, but in terms of general effort (one developer one vote), engineers at twitter, amazon, facebook and google know quite well how to explain the role of the L1 cache, and are extremely knowledgable of concurrency issues. They know the cost of a store fence and a read fence, the cost of a CAS and of a task switch. They tackle latency and scaling issues that those at Blizzard and CCP haven't even dreamed of. They know how to take advantage of garbage-collectors and JITs, and can tell you exactly what machine code the JVM generates in what circumstances. They use read-write locks with striped counters that recognize contention and grow to accomodate it. They choose wait-free queues based on trade-offs between raw performance and garbage generation. They use fork-join, Clojure reducers and immutable data structures. They use parallel data structures. They use Erlang or Akka for actors. Believe me, they know how to reduce latencies to microseconds when they need it, and they need it more than you think. I've even seen someone experiment with eventual consistency at the CPU core level vs. CASs. I don't know about all game companies, but I can assure you that Twitter and Google know how to get more out of each of their cores than Blizzard or CCP (having spoken to Blizzard and CCP engineers). 


Game engines usually aren't just a client-side only thing.
For a single-player game, the client machine has to run both the game client and the game server.


Yes, but the server side, as well as the middleware mentioned by @hplus0603 above, is little more than a networking framework. It's analogous to Apache or Nginx in the web industry, or to RTI in defense. It doesn't help you with scheduling and concurrency. 


A proprietary (read: NIH) one, because the ones we looked at weren't suitable for games.


Of course. :) Nice architecture, though. I especially liked the delayed messages bit. But, correct me if I'm wrong, at the end of each cycle all threads are effectively stopped and only one core is used, right? Also, how do you ensure that each thread has a similar workload? That's fine if you have 8 cores, but what if you have 100? Stopping 100 threads and waiting for stragglers has a huge impact on scalability.


Let me tell you how SpaceBase works internally. Each R-Tree node (a sub-region of space, dynamically resized by the number of objects), is an actor, only fork-join is used for actor scheduling, and it employs work-stealing queues. Work stealing queues ensure that no core ever waits for others: every message produced by an actor goes into a thread-local queue, but when a thread runs out of messages, it steals one from another thread's queue, trying to steal a task (message) that is likely to generate a large number of subtasks (other messages), so that cross-thread stealing wouldn't happen often. 


Determinism can be massively important!
A massive part of game development is spent debugging, and a deterministic sim makes reproducing complex cases very simple -- someone who has produced the error can just save their inputs-replay file. Anyone can then replay this file to see how the error case developed over time, breaking execution and inspecting state at the error itself, but also in the lead up to the error.


It appears that your actor architecture was designed with two core requirements (other than the main ones of using multi-core): developer habits and determinism, and it seems that you've done a great job at satisfying the two, at the cost of some performance and scaling, I guess. Here's how those maligned "web startups" do it in the performance sensitive bits: You have an in-memory cyclical buffer per threads that records events along with a high-resolution timetag (obtaining the timetag is the expensive operation here, but recording is only turned on during debugging). When a bug occurs, the buffers from all threads are merged. 


If your environment protects you from races, you usually don't even need that. All you need to do is replay the particular inputs a single task received.


breaking "frameworks" down into a large collection of simple, single-responsibility libraries, resulting in a much more flexible code-base.


I agree with that wholeheartedly! Unfortunately, when your library does scheduling, it inevitably becomes a "framework"... in most languages: Clojure, because it has abandoned the traditional OO model and implements on a uniform data access pattern, and because it's so beautifully (without impacting practicality) functional, it has the advantageous property of making any framework a library. All objects are the same, you don't need to inherit anything and mixins can be, well, mixed-in if necessary. It also protects you from data races and can be extremely fast. If you don't want frameworks, use Clojure and they will all magically disappear. 



One popular game engine that I've used implemented a pretty neat stream processing system where: ...


Nice! Reminds me of this.


The 'project' was shut down, in terms of funding being withdrawn, but that doesn't mean so much for open-source. The people who worked on it had multi-node up and running, and the contention costs were so high that adding nodes actually reduced performance, hence it wasn't distributed in that state. It turns out that an optimistic locking model degrades significantly when distributed across the network and when the simulation's spatial coherence is not high enough to keep interacting entities colocated, with a resultant cascade effect when entities end up spending more and more of their time on the wire and not in memory.


Maybe an optimistic locking model degrades significantly when distributed across the network and maybe not, but Darkstar (or RedDwarf as it was later called) never tested that. It never reached that stage, and only did simple test distributing their single-node code. Check your sources...


If you see online games as spatial simulations then they are a suitable nail for your spatial database to hammer. But that is one particular reductionist view of games. Much of the game has little or nothing to do with space and doesn't benefit from spatial coherence. Solving the spatial problem well is hard, which you know, and which is why you feel that what you've made has potential worth. But game developers realised that the spatial problem was hard about 10 years ago and designed around it, in a way that suits the rest of their game and which isn't limiting when faced with non-spatial queries.


... and that's a reductionist view of what I outlined in my original blog post. SpaceBase is spatial, but we're working on other products using the same approach which aren't.

You can do the same.


This may seem conservative when compared to the world of websites that make a loss year after year, but the market conditions for these businesses are very different.


Perhaps for good reason game devs are conservative, and that's why we're not trying to sell them our products any more... I posted here because I thought some might be interested in the approach. You never know, an independent game developer might decide to use our tech for a new game and enjoy the performance boost. It's free for small instances anyway.

In Topic: A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceshi...

04 March 2013 - 07:42 AM

I disagree! In my view, shared-state concurrency is very different from a cohesive distributed view. Being able to be divergent but eventually consistent on disparate nodes is a hugely important distinction that you don't have in MESI.


What I meant was the concurrency is essentially distribution across nodes with a relatively slow interconnect between them, just like a network. I don't understand how the two can be considered different. As I said before, all computer architecture is built on the idea of local nodes with relatively slow interconnects. Any use of the comm bus, be it between cores, NUMA sockets or cluster nodes should be minimized in order to increase scaling. The comm bus has two properties that hinder performance: low throughput and low latency. Eventual consistency tries to bypass the latency limitation. However, eventually consistent scaling sometimes sacrifice consistency for no good reason. They are architecturally lazy when it comes to object placement by employing a very simple scheme: usually a DHT. With such a bad strategy, sacrificing consistency gives latency benefits. But smarter object placement schemes can achieve the same (amortized, not worst case) latency while maintaining consistency. 


which is why middleware is so damn popular in the games industry (contrary to your assertion)


Is it (I'm asking seriously)? It is certainly true on the client side with game engines, but which server-side frameworks are in common use in games? 


On that note, with your space-ship example, it's not obvious what kind of ordering constraints are/can-be applied to the different processes. Are all beams fired before any damages are calculated, etc?


The demo was purposefully written to simulate asynchronous requests coming over the network, each being processed as it arrives with no global "ticks".


In the Actor Model implementation that I've used for games, these kinds of ordering issues were explicit to the programmer, allowing them to maintain a deterministic simulation, regardless of the number of cores or the scheduling algorithm used.


The same could be achieved with SpaceBase (though that was not the intention in the demo). You could collect all requests and then process them in parallel, deterministically. I'm not sure this property is always so important (or important at all), though. BTW, what actor implementation have you used? 


most use something closer to a stream-processing or flow-based programming model, which is easy to reason about (you can write the main flow in a single-threaded style and have it be automatically decomposed)


Which frameworks are used for that?


I wouldn't agree with that. Multiple threads sharing the same RAM without restriction gives us the "shared state" model of concurrency. When we build a message passing framework, we build it directly on top of this model.


Of course. Software level messaging abstraction is implemented on top of a shared RAM abstraction, which is, in turn, implemented on top of hardware level messaging. The messages are different, but the point is that shared state is always an abstraction on top of messaging, and computer architecture is such that this cross-"node" messaging should be minimized. We can call it the "shared-state/message-passing duality". Sometimes it's best to work with one abstraction, and sometimes with the other. However, a shared-state framework can certainly improve performance in many cases, as it has more knowledge about the domain semantics to apply optimizations.