Jump to content

  • Log In with Google      Sign In   
  • Create Account


A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceships on My Laptop


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
34 replies to this topic

#21 pronpu   Members   -  Reputation: 139

Like
0Likes
Like

Posted 04 March 2013 - 07:42 AM

I disagree! In my view, shared-state concurrency is very different from a cohesive distributed view. Being able to be divergent but eventually consistent on disparate nodes is a hugely important distinction that you don't have in MESI.

 

What I meant was the concurrency is essentially distribution across nodes with a relatively slow interconnect between them, just like a network. I don't understand how the two can be considered different. As I said before, all computer architecture is built on the idea of local nodes with relatively slow interconnects. Any use of the comm bus, be it between cores, NUMA sockets or cluster nodes should be minimized in order to increase scaling. The comm bus has two properties that hinder performance: low throughput and low latency. Eventual consistency tries to bypass the latency limitation. However, eventually consistent scaling sometimes sacrifice consistency for no good reason. They are architecturally lazy when it comes to object placement by employing a very simple scheme: usually a DHT. With such a bad strategy, sacrificing consistency gives latency benefits. But smarter object placement schemes can achieve the same (amortized, not worst case) latency while maintaining consistency. 

 

which is why middleware is so damn popular in the games industry (contrary to your assertion)

 

Is it (I'm asking seriously)? It is certainly true on the client side with game engines, but which server-side frameworks are in common use in games? 

 

On that note, with your space-ship example, it's not obvious what kind of ordering constraints are/can-be applied to the different processes. Are all beams fired before any damages are calculated, etc?

 

The demo was purposefully written to simulate asynchronous requests coming over the network, each being processed as it arrives with no global "ticks".

 

In the Actor Model implementation that I've used for games, these kinds of ordering issues were explicit to the programmer, allowing them to maintain a deterministic simulation, regardless of the number of cores or the scheduling algorithm used.

 

The same could be achieved with SpaceBase (though that was not the intention in the demo). You could collect all requests and then process them in parallel, deterministically. I'm not sure this property is always so important (or important at all), though. BTW, what actor implementation have you used? 

 

most use something closer to a stream-processing or flow-based programming model, which is easy to reason about (you can write the main flow in a single-threaded style and have it be automatically decomposed)

 

Which frameworks are used for that?

 

I wouldn't agree with that. Multiple threads sharing the same RAM without restriction gives us the "shared state" model of concurrency. When we build a message passing framework, we build it directly on top of this model.

 

Of course. Software level messaging abstraction is implemented on top of a shared RAM abstraction, which is, in turn, implemented on top of hardware level messaging. The messages are different, but the point is that shared state is always an abstraction on top of messaging, and computer architecture is such that this cross-"node" messaging should be minimized. We can call it the "shared-state/message-passing duality". Sometimes it's best to work with one abstraction, and sometimes with the other. However, a shared-state framework can certainly improve performance in many cases, as it has more knowledge about the domain semantics to apply optimizations. 



Sponsor:

#22 hplus0603   Moderators   -  Reputation: 4966

Like
0Likes
Like

Posted 04 March 2013 - 11:20 AM

each being processed as it arrives with no global "ticks"

While the original QuakeWorld did something like this, games have since then moved away from this for several reasons, including repeatability, fairness, debuggability, and determinism.

which server-side frameworks are in common use in games?

If you choose an engine, like UDK, CryEngine, Id Tech, C4, Unity, or whatever, you get a networking system to go with it. Tha one exception I know of is Gamebryo, although a quick Google says they just recently added networking. Interestingly, I think they merged with a MMO technology company a few years ago, and then dropped the MMO server-side part...

It is true that, if you as a game developer do not choose an engine, but instead assemble libraries (a fine methodology IMO,) there are only low-level networking systems (RakNet, Gamespy, Lidgren, etc.)

If you look at RAD Game Tools, they have nice, focused libraries for specific applications. A game-level networking library in that vein would probably be very welcome. Then we're back at "library" vs "framework." If you're picking a "framework" then, in game terms, that's an "engine" and should solve the soup-to-nuts game tech problem.

If you cannot re-package your technology as a library, perhaps the best option is to make common cause with one of the existing game technologies -- be it Hero Engine, Unreal, Id Tech, or Bigworld.
enum Bool { True, False, FileNotFound };

#23 Kylotan   Moderators   -  Reputation: 3329

Like
0Likes
Like

Posted 04 March 2013 - 05:45 PM

At the end of the day you'll have to trust the framework to do the right thing. We take care of the edge cases for you.

 

But this doesn't tally with this:

 

 

SpaceBase only supports spatial queries. Future products will be built for other kinds of queries as well.

 

If you see online games as spatial simulations then they are a suitable nail for your spatial database to hammer. But that is one particular reductionist view of games. Much of the game has little or nothing to do with space and doesn't benefit from spatial coherence. Solving the spatial problem well is hard, which you know, and which is why you feel that what you've made has potential worth. But game developers realised that the spatial problem was hard about 10 years ago and designed around it, in a way that suits the rest of their game and which isn't limiting when faced with non-spatial queries. Unfortunately you're now coming along with a solution to a problem most MMO developers don't really believe they have, and asking them to change everything about how they work in order to take advantage of it. As mentioned above, this is 'high friction'.

 

Auction houses, global chat, guilds, private messages to anybody in the world, a Who list, etc. - these are the real "edge cases" in your system and you don't seem to be giving positive responses to how they would be handled. If SpaceBase does not let me implement them, or forces me to have a parallel database that I must maintain separately, then it is not suitable for MMOs. This is something you'll have to think hard about.

 

 

Project Darkstar was shutdown before reaching the point of a multi-node distribution.

 

The 'project' was shut down, in terms of funding being withdrawn, but that doesn't mean so much for open-source. The people who worked on it had multi-node up and running, and the contention costs were so high that adding nodes actually reduced performance, hence it wasn't distributed in that state. It turns out that an optimistic locking model degrades significantly when distributed across the network and when the simulation's spatial coherence is not high enough to keep interacting entities colocated, with a resultant cascade effect when entities end up spending more and more of their time on the wire and not in memory.

 

 

For some reason, when you show a new technology to someone at, say, Twitter, they'll immediately start thinking how they can take advantage of it.

 

Startup tech companies have almost literally money to burn in the name of research and discovering cool things. Twitter has been run almost entirely on the back of investors throwing money at it in the hope that they find a way to turn a profit later. Not only can they afford to experiment, but arguably they are being paid to do exactly that.

 

Games companies are not tech startups. They are rarely funded by investors in the same sense and are much more linked to retail sales. There's little room for experimentation or risk-taking. Making games takes a lot of time, and most projects are not profitable anyway. As a result, you almost always need to reuse the code you have already invested in. It doesn't make good business sense to throw out half a decade of past work to take a chance on a new language and indeed a new paradigm for your next game when you have firm market evidence showing that customers are happy to play in the type of system that was designed several years earlier. This may seem conservative when compared to the world of websites that make a loss year after year, but the market conditions for these businesses are very different.


Edited by Kylotan, 04 March 2013 - 05:46 PM.


#24 hplus0603   Moderators   -  Reputation: 4966

Like
0Likes
Like

Posted 04 March 2013 - 09:00 PM

I also think there is a lineage difference. Web developers come out of HTML design and high-level application frameworks. They have never had a multi-year project die because of occasional one-second lag spikes. They generally wouldn't be able to explain how a L1 memory cache is different from an OS-level block buffer cache. They want to use whatever new shiny toy gets their site to look and work better for their test user -- and most of them don't really have that many concurrent users. A lot of interesting products and use cases come out of that lineage, but at the end of the day, the successful ones end up having to hire large teams of experienced low-level people to re-write and operate their site at scale. Those experienced guys, at turn, have often seen any number of "promising frameworks" turn operational infrastructure to mush, and are rightfully conservative.

 

Someone who wants to sell dreams to application developers had better speak application developer language, which is all about getting more stuff out with less effort. Someone who wants to sell systems to low-level operational people, in turn, had better be prepared to justify why they're good, like, say, memcached, rather than terrible like, say, Rails.


enum Bool { True, False, FileNotFound };

#25 Hodgman   Moderators   -  Reputation: 27660

Like
1Likes
Like

Posted 04 March 2013 - 09:03 PM


Sorry, I meant that everyone's GPU code is JIT compiled.
You control the GPU via an API like D3D, and shader code (e.g. HLSL) which is compiled into an intermediate bytecode format.

Of course. That's still an AOT compiler, not a JIT, as it compiles the shader before it runs for the first time. I was referring to the merits of a true profiling JIT, one that performs optimizations based on the running profile of the code. I thought that somehow you've done that with shaders.


The actual code that's run on the GPU depends on both the shader bytecode and the command stream generated from the API.
e.g. when compiling shader bytecode, the driver might have a choice to optimize for reduced register usage (size), or reduced ALU operations (speed). If the compiler guesses that this shader will be memory-bound because it contains many fetch instructions, then it may choose to optimize for size over speed, because this will allow more threads to be in flight at once (the GPU's version of hyperthreading is flexible in the number of "hardware threads" that can be run, depending on the register usage of the current shader).
However, whether this guess is correct or not depends on the command stream. The API can configure the inputs such that all of the memory fetches will read the same byte of memory, resulting in an epic caching win and no memory bottleneck, or it can configure the inputs such that every memory access is random an uncacheable. If the former, the compiler should have optimized for speed, and if the latter it should have optimized for size.
There's many other examples like this -- e.g. when two different shader programs are executing simultaneously on different sub-sets of the GPUs resources, the best optimization for a shader method may depend on what the "neighbouring" shader is.
Apart from these kinds of JIT-to-optimize cases, there's also times when the driver is outright required to modify/recompile the code -- e.g. depending on what kind of texture you've bound via the API, the instructions to fetch and decode floating point values from it will differ (e.g. for byte-textures you need a "divide by 255"); some GPUs have specialized hardware for this that are configured by the API, while others require the shader assembly to be recompiled with the appropriate instructions based on the API state.
 

Is [middleware actually popular] (I'm asking seriously)? It is certainly true on the client side with game engines, but which server-side frameworks are in common use in games?

Game engines usually aren't just a client-side only thing.
For a single-player game, the client machine has to run both the game client and the game server.
In a peer-to-peer game, the same is true, as often every client is acting as a shared server.
In a client-server game, a popular feature is to allow one player to act as a server, instead of requiring a dedicated server machine.

Given this, it's helpful if you can just write the game once, in such a way where it can be used in all of the above models, by simply changing some replication and authority policies. High-end game engines contain a lot of utilities to help write these kind of "simultaneously client-only/server-only/client-is-server" code-bases.
So game engines are both client middleware and server middleware (and often you'll supplement an engine with a lot of extra middleware -- the big engines often provide integration/glue code for other popular middleware packages to make it easy to use them alongside their engine).
 
Often as well as the game client and server, there is a "master server", which players connect to first in order to find a particular server.
e.g. a game server hosts 64 players at once and runs a game simulation, while the master server hosts millions of players at once, but only runs chat and matchmaking logic.
In this space, you'll definitely find a lot of standard technologies like MySQL, memcached, JSON, REST API's over HTTP, Comet/Push web stuff, MongoDB, etc...
 
MMOs are similar to the latter group, but I've never worked for an MMO developer so I can't comment on their tech.

In the MMO-space, I know of quite a few that are largely written in Python (e.g. EVE, any Big World ones), and this has come back to bite them in the age of concurrency ("global interpreter lock"...). Eve got blasted recently for being mainly single-core, and part of their official response said (paraphrasing) that multi-core isn't just something we can switch on, and python makes the transition especially hard on us. So those stories do fit with your anonymous MMO anecdote about finding it hard to move away from the single-threaded game-loop model wink.png
MMOs have largely been confined to the PC realm though, so unlike console developers, they weren't forced to make the multi-core transition back when multi-core consoles became mainstream.
 

BTW, what actor implementation have you used?

A proprietary (read: NIH) one, because the ones we looked at weren't suitable for games wink.png
It worked by:
* It operated in 'cycles' where a groups of messages were executed, which would result in objects being created/destroyed/being-sent-messages. Each sim frame, the actor model would continue to run these cycles continuously until no more messages were produced, at which point it would begin the next sim frame.
* Every thread had a thread-local buffer where messages could be written. Whenever you called a method on an actor, the args would be written to the current thread's buffer and a future returned.
* Futures didn't have any kind of complex waiting logic -- the system knew immediately how many cycles in the future the value would be ready. The only way to 'resolve' a future into a value was to use it as an argument to a message. That message would be delayed the appropriate number of cycles so that it wasn't executed until after it's arguments existed.
* At the end of each cycle:
*** A new buffer full of messages has been produced by each thread. These were merged into one list, and sorted by actor-ID (and a 2nd sorting key to provide determinism via message priority). That sorted queue was then split into one non-overlapping range of actors per thread, to be executed next cycle.
*** Any objects who's reference count had been decremented would be checked for garbage collection / destruction.
*** Any new objects would have their IDs assigned in a deterministic manner.
 
It was nice simply because it let people, for the most part, continue writing in a style similar to the traditional OOP that they were used to, but the execution would be split across any number of cores (up to the number of active Actors), while still remaining completely deterministic, and with very little synchronisation between threads (most communication was wait-free in optimal conditions, even reference-counting was wait-free and without cache-contention).
 

I'm not sure this property [determinism] is always so important (or important at all), though.

Determinism can be massively important!
A massive part of game development is spent debugging, and a deterministic sim makes reproducing complex cases very simple -- someone who has produced the error can just save their inputs-replay file. Anyone can then replay this file to see how the error case developed over time, breaking execution and inspecting state at the error itself, but also in the lead up to the error.
 
Many games rely on deterministic simulation in order to even be feasible from a networking point of view. e.g. RTS games are pretty much the only genre where you can actually have several thousand player controlled entities interacting in a world. The usual methods of state-replication produce ridiculously high bandwidth requirements when applied to these situations, so instead, it's extremely common for multiplayer RTS games to instead use a lock-step simulation where all the players only share their inputs.
The bandwidth between keyboard and chair is very low, so networking based on this input means that you can have an unlimited number of units on the battlefield with a fairly constant network bandwidth requirement. Players simply buffer and share their inputs, and then apply them at agreed upon times, resulting in the same simulation on all clients.

Also, many console games don't have the luxury of dedicated servers. Instead, they often use the one-client-is-the-server model. This means that the server's outward bandwidth is restricted to whatever the upload bandwidth is on a residential DSL connection (which is pretty low). This makes networking strategies based around determinism and input sharing quite popular, due to the low bandwidth requirements.

For a general-purpose game engine that can be used for any genre, it's pretty much required to support determinism in case the end user (game developer) requires it. To not support it, is to cut off a percentage of your clients.

[edit missed this one]

Which frameworks are used for [stream processing]?

I'm not aware of any popular stream processing middleware, the ones I've been exposed to have either been a component of a game engine, or a home-grown proprietary solution. I guess one of the nice things about stream processing is that you don't need a big complicated framework to implement it, so the damage of NIH isn't as bad.


On a tangent, the difference between (pejorative) "frameworks" and (simple) "libraries" is an interesting one. Technically, they're often both just a code library, but a "framework" usually requires that all the code that you write when using it conforms to it's particular worldview. e.g. every class has to follow some pattern, inherit some base, provide some overload, be mutated in some certain way, etc, etc... You can usually tell when you try and use two different "frameworks" together, and end up in a hell of a conflicting mess. Game engines often fit this category -- they take on so many responsibilities that everything you do is tightly coupled to their way of doing thigs...
In some of the projects that I've worked on, these kinds of frameworks have caused a lot of internal friction, so I've seen many teams resort to attacking code-bases with a figurative axe, breaking "frameworks" down into a large collection of simple, single-responsibility libraries, resulting in a much more flexible code-base.
The reason I bring this up was because the last time I saw this, the instigator was a parallel processing framework, and the need for some "exceptions" to be bolted onto it. Like you said earlier, even functional languages sometimes resort to using the shared-state model! The end result was that we ended up with a few different processing libraries that could be used together easily, without any frameworks tying code down too strongly to a single model.


On the PS3, there is a popular bit of middleware called SPURS, which basically implements a "job" type system -- a job is a function and a bunch of parameters/data-pointers, which you can put into a queue and can later check/wait for it's completion.
Because of this, many game engines use this model as their low level concurrency layer (implementing "job" systems on 360/PC/etc as well). They then often build another layer on top of this job system that allows for stream processing or other models.

One popular game engine that I've used implemented a pretty neat stream processing system where:
* the processing pipeline could be described as data - the kernels would be compiled from this description dynamically.
* the pipeline description was a graph of nodes representing function pointers, and the edges told it how the function's input/output arguments flowed between nodes.
* when compiling, you could tell it how much 'scratch memory'/'node memory' etc it had available, which would affect how it unrolled the loops and constructed kernels etc...
** e.g. maybe it will just call each function in sequence on one bit of data at a time, or maybe it will call function #1 on 64 bits of data then function #2 using those 64 inputs, etc...
* on platforms that allowed for code-generation, it would actually memcpy code around the place to create highly efficient kernels. On other platforms it operated by actually calling function pointers.

You can read a bit about a similar system here: http://www.insomniacgames.com/tech/articles/0907/files/spu_shaders_introduction.pdf

Edited by Hodgman, 05 March 2013 - 02:39 AM.


#26 pronpu   Members   -  Reputation: 139

Like
0Likes
Like

Posted 05 March 2013 - 04:59 AM

The actual code that's run on the GPU depends on both the shader bytecode and the command stream generated from the API.

 

Cool. I actually did not know that.

 

I also think there is a lineage difference. Web developers come out of HTML design and high-level application frameworks. They have never had a multi-year project die because of occasional one-second lag spikes. They generally wouldn't be able to explain how a L1 memory cache is different from an OS-level block buffer cache. They want to use whatever new shiny toy gets their site to look and work better for their test user -- and most of them don't really have that many concurrent users.

 

I'm afraid this is what many game developers believe but it is patently false. True, most web companies (one company one vote) match your perception, but in terms of general effort (one developer one vote), engineers at twitter, amazon, facebook and google know quite well how to explain the role of the L1 cache, and are extremely knowledgable of concurrency issues. They know the cost of a store fence and a read fence, the cost of a CAS and of a task switch. They tackle latency and scaling issues that those at Blizzard and CCP haven't even dreamed of. They know how to take advantage of garbage-collectors and JITs, and can tell you exactly what machine code the JVM generates in what circumstances. They use read-write locks with striped counters that recognize contention and grow to accomodate it. They choose wait-free queues based on trade-offs between raw performance and garbage generation. They use fork-join, Clojure reducers and immutable data structures. They use parallel data structures. They use Erlang or Akka for actors. Believe me, they know how to reduce latencies to microseconds when they need it, and they need it more than you think. I've even seen someone experiment with eventual consistency at the CPU core level vs. CASs. I don't know about all game companies, but I can assure you that Twitter and Google know how to get more out of each of their cores than Blizzard or CCP (having spoken to Blizzard and CCP engineers). 

 

Game engines usually aren't just a client-side only thing.
For a single-player game, the client machine has to run both the game client and the game server.

 

Yes, but the server side, as well as the middleware mentioned by @hplus0603 above, is little more than a networking framework. It's analogous to Apache or Nginx in the web industry, or to RTI in defense. It doesn't help you with scheduling and concurrency. 

 

A proprietary (read: NIH) one, because the ones we looked at weren't suitable for games.

 

Of course. :) Nice architecture, though. I especially liked the delayed messages bit. But, correct me if I'm wrong, at the end of each cycle all threads are effectively stopped and only one core is used, right? Also, how do you ensure that each thread has a similar workload? That's fine if you have 8 cores, but what if you have 100? Stopping 100 threads and waiting for stragglers has a huge impact on scalability.

 

Let me tell you how SpaceBase works internally. Each R-Tree node (a sub-region of space, dynamically resized by the number of objects), is an actor, only fork-join is used for actor scheduling, and it employs work-stealing queues. Work stealing queues ensure that no core ever waits for others: every message produced by an actor goes into a thread-local queue, but when a thread runs out of messages, it steals one from another thread's queue, trying to steal a task (message) that is likely to generate a large number of subtasks (other messages), so that cross-thread stealing wouldn't happen often. 

 

Determinism can be massively important!
A massive part of game development is spent debugging, and a deterministic sim makes reproducing complex cases very simple -- someone who has produced the error can just save their inputs-replay file. Anyone can then replay this file to see how the error case developed over time, breaking execution and inspecting state at the error itself, but also in the lead up to the error.

 

It appears that your actor architecture was designed with two core requirements (other than the main ones of using multi-core): developer habits and determinism, and it seems that you've done a great job at satisfying the two, at the cost of some performance and scaling, I guess. Here's how those maligned "web startups" do it in the performance sensitive bits: You have an in-memory cyclical buffer per threads that records events along with a high-resolution timetag (obtaining the timetag is the expensive operation here, but recording is only turned on during debugging). When a bug occurs, the buffers from all threads are merged. 

 

If your environment protects you from races, you usually don't even need that. All you need to do is replay the particular inputs a single task received.

 

breaking "frameworks" down into a large collection of simple, single-responsibility libraries, resulting in a much more flexible code-base.

 

I agree with that wholeheartedly! Unfortunately, when your library does scheduling, it inevitably becomes a "framework"... in most languages: Clojure, because it has abandoned the traditional OO model and implements on a uniform data access pattern, and because it's so beautifully (without impacting practicality) functional, it has the advantageous property of making any framework a library. All objects are the same, you don't need to inherit anything and mixins can be, well, mixed-in if necessary. It also protects you from data races and can be extremely fast. If you don't want frameworks, use Clojure and they will all magically disappear. 

 

 

One popular game engine that I've used implemented a pretty neat stream processing system where: ...

 

Nice! Reminds me of this.

 

The 'project' was shut down, in terms of funding being withdrawn, but that doesn't mean so much for open-source. The people who worked on it had multi-node up and running, and the contention costs were so high that adding nodes actually reduced performance, hence it wasn't distributed in that state. It turns out that an optimistic locking model degrades significantly when distributed across the network and when the simulation's spatial coherence is not high enough to keep interacting entities colocated, with a resultant cascade effect when entities end up spending more and more of their time on the wire and not in memory.

 

Maybe an optimistic locking model degrades significantly when distributed across the network and maybe not, but Darkstar (or RedDwarf as it was later called) never tested that. It never reached that stage, and only did simple test distributing their single-node code. Check your sources...

 

If you see online games as spatial simulations then they are a suitable nail for your spatial database to hammer. But that is one particular reductionist view of games. Much of the game has little or nothing to do with space and doesn't benefit from spatial coherence. Solving the spatial problem well is hard, which you know, and which is why you feel that what you've made has potential worth. But game developers realised that the spatial problem was hard about 10 years ago and designed around it, in a way that suits the rest of their game and which isn't limiting when faced with non-spatial queries.

 

... and that's a reductionist view of what I outlined in my original blog post. SpaceBase is spatial, but we're working on other products using the same approach which aren't.

You can do the same.

 

This may seem conservative when compared to the world of websites that make a loss year after year, but the market conditions for these businesses are very different.

 

Perhaps for good reason game devs are conservative, and that's why we're not trying to sell them our products any more... I posted here because I thought some might be interested in the approach. You never know, an independent game developer might decide to use our tech for a new game and enjoy the performance boost. It's free for small instances anyway.



#27 Kylotan   Moderators   -  Reputation: 3329

Like
0Likes
Like

Posted 05 March 2013 - 07:53 AM

Maybe an optimistic locking model degrades significantly when distributed across the network and maybe not, but Darkstar (or RedDwarf as it was later called) never tested that. It never reached that stage, and only did simple test distributing their single-node code. Check your sources...

 

Here's a video of them showing multi-node operation at GDC Austin:

 

On this forum thread it was said, "the Sun team showed a working prototype of the multi-node functionality at GDC-Austin before Oracle disbanded the team.  [...] We are working on a simpler solution at Nphos that we are confident will allow us to scale horizontally for casual games, and *may* perform well enough for MMORPGs". The very phrase "may perform well enough for MMOs" gives an insight into how poorly the system was scaling before and how low their expectations were! The horizontal case for casual games is already well-provided for by other tech anyway, which is probably why Nphos doesn't seem to be around any more.

 

Perhaps for good reason game devs are conservative, and that's why we're not trying to sell them our products any more... I posted here because I thought some might be interested in the approach.

 

It's certainly interesting! I'm sorry if I've come across as argumentative in trying to show that there would be problems using this tech for MMOs, as I assumed that was why you were posting here. I'm quite interested in the load-balancing and spatial coherence aspects myself, as someone who's working on distributed seamless world tech for MMOs. I also have experience of working on a team which shipped an MMO and came to appreciate that while features that buy you scalability are great, ease of use for the programmers is probably more important in terms of getting the project finished and working. Ultimately players are far more tolerant of shards that are half the size than single worlds with bugs. :)



#28 pronpu   Members   -  Reputation: 139

Like
0Likes
Like

Posted 05 March 2013 - 08:36 AM

The very phrase "may perform well enough for MMOs" gives an insight into how poorly the system was scaling before and how low their expectations were! The horizontal case for casual games is already well-provided for by other tech anyway, which is probably why Nphos doesn't seem to be around any more.

 

Yes, that's referring to Nphos's simple solution.As for Darkstar -- there was a prototype, but research was far from finished. At no point in the project, throughout its various incarnations, have they released even an alpha version of multinode. They believed they would be able to achieve terrific scaling, but all efforts on multinode were halted once Oracle defunded the project.

 

there would be problems using this tech for MMOs, as I assumed that was why you were posting here.

 

Yes, there would be problems, but also some great benefits to be gained. Nevertheless, we've given up all efforts to persuade game developers of anything. Turing and von Neumann themselves could not have persuaded a game developer. But the lucky developer who will try our stuff will get some great results, and we'll be all the more happy for him. That's not our business plan, though. If you give it a try you'll be happy that you did; if not -- well, I don't try everything, either.

 

ease of use for the programmers is probably more important in terms of getting the project finished and working.

 

Yes, but ease of use is different from sustaining habits. You're talking about habits, not ease, and, like we both said, game developers are conservative :) Many "web developers that don't know what a TLB is" were able to quickly wrap their heads around Clojure, Erlang, Go and Node and mostly reaped the benefits of better productivity, even at the price of changing their habits. But, to each his own.

 

Ultimately players are far more tolerant of shards that are half the size than single worlds with bugs.

 

Yes, because they only know what they're getting. I'd take that further. Players are more tolerant of games that haven't progressed much in terms of features and abilities in something like a decade; this particularly applies to MMOs. Google and Facebook are able to process in realtime several orders of magnitude more than they were a decade ago. MMOs operate at pretty much the same scale. 

 

If you'd care to hear my thesis about it, here it is, and it's completely different from yours, about risk and investments. The reason for conservatism is that AAA games, especially MMOs, are prohibitively expensive to produce, and the software development is mere fractions of the cost. It almost all goes to content. For this reason, incumbents like Blizzard and CCP have little to fear from small startups with a better software architecture threatening their position. A startup could build an MMO architecture far superior to Blizzard's, but it doesn't matter because they won't have the resources to produce as much content and compete with WoW or EVE. There are very few incumbents, all of them co-existing happily together. And that is why the big studios have little reason to constantly innovate when it comes to architecture. To push MMO studios to innovate you'd have to come up with a game that raises player expectations, and that would only happen if players like it, which, in turn, will only happen if the content is rich. So it won't come from a startup. Without this expectation from their customers, innovation happens at a glacial pace. No one is in a hurry. 

 

This is completely different in industries where you don't have such high content production costs, and so even the incumbents are constantly fearful of newcomers; so they must innovate. Engineers there are constantly looking for new techniques and approaches that would give them an edge over the competition. 



#29 Kylotan   Moderators   -  Reputation: 3329

Like
0Likes
Like

Posted 05 March 2013 - 11:06 AM

At no point in the project, throughout its various incarnations, have they released even an alpha version of multinode.

 

Sure, but 'not released' is not the same as 'not implemented'. They have had it up and running and demonstrated it working. But the developers have said that they needed to do more research to get it scaling upwards rather than downwards and the person close to the project that went on to form Nphos had rather low expectations based on his experiences so far.

 

You're talking about habits, not ease, and, like we both said, game developers are conservative smile.png

 

Hmm, I would say that writing a sequential script where one entity can find and access another entity directly is easier than factoring it out into a selection of queries and callbacks. Even if you had a language which let you write it in a more fluent way, eg. hiding the callbacks via coroutines, you've still added a layer of implicit concurrency that the developer now has to worry about. Yes, it's comparing apples with oranges, because your system scales out in a way the existing systems don't, but you don't get that for free unless you were already doing things the hard way. And you've pretty much admitted that there are key MMO features that just can't be done with your system, which makes it a hard thing to sell to a game designer.

 

MMOs operate at pretty much the same scale [as a decade ago]

 

Certainly true, for the most part. It's lamentable that the most popular MMO is one of the least technologically advanced when it comes to scalability. But it's hard to make an argument that there's a big demand to have everybody in one large game world. In fact there are some game design reasons why people do actually want separate servers, and indeed separate instances within servers.

 

The reason for conservatism is that AAA games, especially MMOs, are prohibitively expensive to produce, and the software development is mere fractions of the cost. It almost all goes to content.

 

Content is expensive but the software engineering side is never a mere fraction of the cost, unless you want to count between 2/5 and 3/5 as 'mere'. :) This may or may not change for MMOs after release when mostly it's about adding new content, but that new content also requires code support.

 

It is true that it is hard to disrupt a market that is full of such large incumbents. But the scalability of the world isn't likely to do it. If "one massive shared space" was a game-changer (no pun intended) then EVE would be far bigger than it is. Better technology only makes a difference where the user experience changes significantly as a result, and the majority of the paying customers in the MMO world seem to be asking to play with increasingly smaller and smaller groups of people in order to better control their play experience. Thus the lure of the seamless world has lost its lustre somewhat. It's easy to mistake games as being a symptom or a demonstration of technology, but really the tech is the tool here, not the product.



#30 pronpu   Members   -  Reputation: 139

Like
0Likes
Like

Posted 05 March 2013 - 11:41 AM

Better technology only makes a difference where the user experience changes significantly as a result, and the majority of the paying customers in the MMO world seem to be asking to play with increasingly smaller and smaller groups of people in order to better control their play experience. Thus the lure of the seamless world has lost its lustre somewhat. It's easy to mistake games as being a symptom or a demonstration of technology, but really the tech is the tool here, not the product.

 

I absolutely agree, yet expectation or demand is often a function of what's available. I think it was Henry Ford who said, "if I were to ask my customers what they wanted they'd say they want a faster horse". I don't think there was demand for video games before there were any, or for 3D shooters before Wolfenstein. New technology just opens new horizons for the more adventurous, imaginative early adopters. Then it needs a killer app for the late adopters to realize its potential.

 

I don't think EVE's success, or its amount, is simply a function of the architecture. I'm sure that if someone came up with a great game that employs a seamless world, a lot of others will follow. Still, that game would have to be great for many reasons; obviously just having a seamless world won't be enough.



#31 Hodgman   Moderators   -  Reputation: 27660

Like
0Likes
Like

Posted 05 March 2013 - 07:21 PM

Yes, but the server side, as well as the middleware mentioned by @hplus0603 above, is little more than a networking framework. It's analogous to Apache or Nginx in the web industry, or to RTI in defense. It doesn't help you with scheduling and concurrency.

I gave you several examples of how they do include scheduling/concurrency frameworks.... They usually provide a large amount of spatial partitioning tools too.

Stuff like your join queries (for example, return all pairs of objects that intersect one another, or return all pairs of objects that are within a given distance from one another) exists in every engine and is highly optimized. In terms of spatial querying, you are competing against technology that most game-devs already have. In terms of parallel processing, many game devs already have solutions, but yes, there's still many trying to transition, who may be potential customers.

 

But, correct me if I'm wrong, at the end of each cycle all threads are effectively stopped and only one core is used, right? Also, how do you ensure that each thread has a similar workload? That's fine if you have 8 cores, but what if you have 100? Stopping 100 threads and waiting for stragglers has a huge impact on scalability.

Good catch. Yes, in that incarnation of the system, every thread had to complete a cycle before the next once could be scheduled, and no thread could start the next cycle until scheduling was complete. Side note: the system that replaced this one mitigated this by scheduling messages to be executed on extra cycle in the future, which allowed two different cycles to be overlapping slightly.

This wasn't as impactful as it seems, because the actor system is only a very small part of the game's frame. There were many other systems running on the "job" model that I mentioned, so whenever a thread was "waiting", instead of going to sleep, it would pop items from a shared job queue and make itself useful. As long as other systems, like physics, etc, were kicked off before starting the actor-processing for a frame, the stalls were filled with other useful work.

Load balancing wasn't much of an issue because the bulk of the workload was in the job system, but instead of splitting a cycle's actor queue into equal pieces per thread, you can split it into a larger number of pieces in a shared queue that idle threads can 'steal' from.

 

Nevertheless, we've given up all efforts to persuade game developers of anything. Turing and von Neumann themselves could not have persuaded a game developer.

You're being silly and ignorant again. I don't understand how you can be refuting stereotypes about X-devs while spreading stereotypes about Y-devs at the same time.

I could crack out the typical Java programmer stereotypes about being able to create a class that abstracts your own mother while being unable to even add two numbers without 18 indirect utility layers, but that's not at all helpful...

 

The ability to convince someone to use your product depends on who they are. I can tell you straight-up that console developers will be a very hard sell, simply because Java doesn't exist there! You could have the greatest middleware in the world, but it's no use if they can't run it. You can't blame headstrong developers with ingrained habits for this.

That leaves PC games... but only PC games that don't also include a console version, because they'll be the same code-base.

If a PC game isn't also on a console, then it's probably either an indie game, or an MMO.

 

Indies have more freedom to experiment with unproven technology and unconventional languages (Java never got a foothold in mainstream games... C# is only starting to make headway now... GC languages are hard to use effectively in RT environments, etc), so they might be a target, especially if you've got a free "starter" license. Hopefully one will become the next Minecraft and be able to afford a real license.

 

MMO's are a more obvious target, because they actually require large-scale server infrastructure, and have the freedom to explore traditional large-scale server-side technologies, so a requirement on the JRE is more welcome. However, the number of large-scale MMO developers in the world is very low, so the chance of netting customers is also pretty low.



#32 Kylotan   Moderators   -  Reputation: 3329

Like
0Likes
Like

Posted 05 March 2013 - 08:10 PM

I gave you several examples of how they do include scheduling/concurrency frameworks.... They usually provide a large amount of spatial partitioning tools too.

Stuff like your join queries (for example, return all pairs of objects that intersect one another, or return all pairs of objects that are within a given distance from one another) exists in every engine and is highly optimized. In terms of spatial querying, you are competing against technology that most game-devs already have. In terms of parallel processing, many game devs already have solutions, but yes, there's still many trying to transition, who may be potential customers.

 

When I last looked into this, there wasn't any middleware that combined spatial queries and parallel processing in the way that SpaceBase attempts to do. Lots of tech can shuffle actors (or zones containing actors) from one node to another, whether automatically or to meet specific demands, and lots of tech can query all actors known to the current node that match some sort of spatial criteria. But I don't know any that can query actors across all nodes based on spatial criteria, and return a reference to actors that can be manipulated (and not just read). This transparent and efficient clustering was the holy grail that Project Darkstar failed to achieve. By comparison, SpaceBase may well manage this, but it apparently comes at a high price: increased code complexity for each task, and certain features made impossible within the framework. The latter point is especially pertinent because it's likely that those are the features that Darkstar supported but which caused its performance problems when they started testing multi-node capability.

 

If you're in a position to either completely rule out or seriously limit any operations that don't take advantage of spatial coherence, partitioning your data becomes much cheaper.



#33 Kylotan   Moderators   -  Reputation: 3329

Like
0Likes
Like

Posted 05 March 2013 - 08:24 PM

The ability to convince someone to use your product depends on who they are. I can tell you straight-up that console developers will be a very hard sell, simply because Java doesn't exist there! You could have the greatest middleware in the world, but it's no use if they can't run it. You can't blame headstrong developers with ingrained habits for this.

That leaves PC games... but only PC games that don't also include a console version, because they'll be the same code-base.

 

Not to mention:

  • Developers want to leverage their heavy past investment in their own existing code that may have no relationship to MMO tech. (eg. Asset loading, audio, rendering, input, scripting languages, game-oriented mathematics.
  • They often want to share code between server and client - so that legacy code, probably in C++, will influence their choice of server language
  • The more senior game developers may only have C++ experience (at least in terms of which languages they are considered expert in) and will be significantly less productive when having to learn new paradigms and new libraries.

I never saw a project that was started completely from scratch before - everything was built in terms of the company's previous code base. But that wasn't just 'habit' - it was the most effective way to get to where they needed to be given the resources they had with the least amount of risk. Obviously there are often more efficient ways to work, but being twice as efficient isn't enough if you have five times more work to do.



#34 Hodgman   Moderators   -  Reputation: 27660

Like
0Likes
Like

Posted 05 March 2013 - 08:25 PM

When I last looked into this, there wasn't any middleware that combined spatial queries and parallel processing in the way that SpaceBase attempts to do.

Oh yeah, I didn't mean to imply otherwise.

 

But within the realm of single-PC games (not distributed servers), these kinds of multi-node concerns don't exist (or with the NUMA PS3, the nodes are so small extremely that data structures are never resident - everything is transient) and existing tech does a decent job of spatial queries. e.g. I can batch up a collection of spatial queries and pass them to PhysX, which will process them over many cores using smart partitioning techniques.


Edited by Hodgman, 05 March 2013 - 08:28 PM.


#35 pronpu   Members   -  Reputation: 139

Like
0Likes
Like

Posted 06 March 2013 - 03:11 AM

In terms of spatial querying, you are competing against technology that most game-devs already have. In terms of parallel processing, many game devs already have solutions, but yes, there's still many trying to transition, who may be potential customers.

 

I don't think so. The whole approach, as outlined in the blog post, is that scalability comes from making parallelization and querying one and the same. If you say you have one or the other, or even both separately -- that's nothing to do with what we provide. We say that IF you have real-time db queries AND the db uses knowledge about the domain and the queries in order to parallelize your business logic THEN you get significant scaling. We're not trying to solve the problem of querying spatial data. We're solving the problem of scaling by utilizing the database. 

 

The main point of our approach is this: we're not utilizing parallelism to speed-up database queries. We're utilizing the database to parallelize (and speed up) your business logic

 

 

You're being silly and ignorant again.

 

Yeah, I'm probably being silly, but I don't think I'm ignorant. smile.png After spending a lot of times talking to potential customers in various markets, some patterns emerge.

 

 

I can tell you straight-up that console developers will be a very hard sell, simply because Java doesn't exist there!

 

All our middleware is server-side only. That's where you find the scaling issues we're trying to solve.

 

However, the number of large-scale MMO developers in the world is very low, so the chance of netting customers is also pretty low.

 

True. That's another reason why we're not trying harder to convince game devs to buy our stuff. For indie developers our software is free anyhow (depending on the game world size).


Edited by pronpu, 06 March 2013 - 04:50 AM.





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS