Quote:Original post by jeffpk

I honestly don't understand? Why are you upset? You are upset that I am assuming a depth of knowledge in a specific specialty in at least some of the posters that is *not* typical of the greater game industry?

Pardon me but I thought that was a compliment.

*shrug*

No, you are assuming posters are lacking a depth of knowledge in an area (MMO back-end systems) with no other evidence than the fact that they disagree with your particular implementation.

Edit: And let me state that I think the implementation does meet the goals of the project, and that it is fine for hobbyist development.

Quote:Original post by swiftcoder
Not *everyone* on this board does have a working knowledge of MMO back-end subsystems. I certainly wouldn't claim to be an expert therein...

But aside from that, Jeff's statement was clearly couched in terms of describing the intended audience of Darkstar (which is *not* aimed solely at MMO development veterans/experts)

While this is true, he has used that as a smokescreen to avoid, ignore, or otherwise misstate the valid criticisms of those who do have MMO or simialr experience posting in this thread, particularly when claiming bias against someone who is in the best position to make valid criticisms.

Additionally, statements like these:

Quote:Something that might be confusing some of you is that I have feeling some of you
are very familiar with enterprise database systems. Thus your first reaction to the DataStore portion of the PDS is "where are all the controls I'm used to" "how can I mess with the transactions", "where are my concurrency controls," "wheres my SQl interface" etc.

Are somewhat frustrating to deal with. This is a misrepresentation of the actual arguments presented, and brushes off the contributions of people who believe (from experience) that these concepts are indeed important in a proper MMO back-end.

Well, i hope this doesnt put your nose furter out of joint.

But the fact of the matter is that enterprise developers have and do use database back-ends VERY different from MMO developers.

All MMOs I know of (and I've worked directly with a number of groups building them over the years) keep their primary game state datastore in non-transactional in-memory systems. These are the direct result of many bugs, including the infamous "dupebugs" which in DB terms are breaks in referential integrity.

As a matter of fact, i had the opportunity a few years ago to ask the lead designer on DAOC how many of their bugs were due to the non-transactional storage of their data and his answer was "almost all of them."

In general MOSt MMOs I know of use the database only to persist out user and billing data. (An exception, btw is EVE. Thats a very interesting design but very specific to their game.) In general the feeling is that databases are not fast enough in response to hold game state... and thats true of traditional database. (There are some interesting variations out there that have come to eb in recent years. TimesTen made an in memory SQl database that was limited in some ways but quite perky, oracle bought them and is selling them now as a front end to oracle.)

Enterprise developers, by contrast, do everything in ACID (Atomic, Isolated, Consistent and Durable) transactions. This ensures the integrity and reliability of their data. But it has a cost in performance. In addition, the performance metrics are different. Enterprise systems are generally worried about total throughput, not individual transaction responsiveness.

Additionally traditional enterprise scalability is based on replication. This works well for enterprise data which is 80% read/20% write but would fail in a game situation where about 50% of your accesses are write accesses.

These were some of the issues we faced when we started designing the PDS. We wanted all the reliability, fault-tolerance and scalability of an enterprise system but to make it easy to use and perform with response times suitable to near-real time apps (which is the technical CS category games fall into.)

Sure, you can handle a lot of these issues by forcing limits on your game designers like static unchanging environments and dividing your game up into Zones... but the former really limits the game design and the latter wastes computer power the operator is paying for in unpopular zones while forcing usership limits in popular ones. Amd none of these techniques catch and deal with the race conditions that result in things like dupe bugs and disappearing items.

The goal of the PDS is to create a truly distributed system for game development that gives you maximal game design flexability and uses ALL your resources effectively while providing enterprise like reliability.

Because, in the end, as a game developer the game state really IS your enterprise.

Quote:Original post by arbitusainst someone who is in the best position to make valid criticisms.

Additionally, statements like these:

Quote:Something that might be confusing some of you is that I have feeling some of you
are very familiar with enterprise database systems. Thus your first reaction to the DataStore portion of the PDS is "where are all the controls I'm used to" "how can I mess with the transactions", "where are my concurrency controls," "wheres my SQl interface" etc.

Are somewhat frustrating to deal with. This is a misrepresentation of the actual arguments presented, and brushes off the contributions of people who believe (from experience) that these concepts are indeed important in a proper MMO back-end.

What makes you think it was a response to any particular question? Are there quotes there?

Honestly you have me thoroughly perplexed. I saw what looked like a potential for confusion and explained that area. If you weren't confused to start with, then why is it even relevant to you?

If you have a particular question then please ask it and I'll do my best to answer it.

Edit: if the question is "how do I store data that I do need to query relationally", I believe I already answered that above. If i didn't please tell me what I missed.

Edit 2: If your response is really "well i don't believe you can do what you set out to do", that's a fair response, but its just a statement of belief. No more no less. And I think we have already, and will continue to, pile up evidence to the contrary. That was one of the reasons we built Snowman, as a clear demonstration that the model can work. Is it a $30 million dollar MMO/ of course not. But is it a real, functional action game that proves out the fundamentals of the design, I believe it is.

If your question is "why should we believe you can make this work", well that's where credentials start to matter some, because there really is no other way to predict the future.

Quote:Original post by jeffpk

All MMOs I know of (and I've worked directly with a number of groups building them over the years) keep their primary game state datastore in non-transactional in-memory systems.

Enterprise developers, by contrast, do everything in durable ATOMIC transactions.

These are not entirely true statements. MMOs are more likely to use a non-transactional model and enterprise developers are more likely to use a transactional model, but this is more of an implementation detail. MMOs will not use databases for game state management, but will use databases for game information collection. The in-memory model will offloaded to a data store at intervals and save off for use in data-mining and reporting. This is integral in cheat detection, verifying player requests/petitions, detecting data errors, detecting runtime errors, monitoring game events/encounters for fine tuning, etc. This information in invaluable when evaluating an MMO from top to bottom and making informed decisions on how to both improve gameplay mechanics and fix existing issues.

This is the source of much of the criticism of the backing store of Darkstar and it is valid. However, the statements that you made are true: Darkstar's goal is not to be the most robust, industry-breaking MMO back-end. It is meant to be a solid solution for scalable online projects, targeted mostly at hobbyists. But you don't have to attack the credibility and the expertise of those who are questioning the applicability of Darkstar in certain situations. There are people here who have a solid understanding of MMO and related project datastore, not simply enterprise developers. Acknowledging that the functionality is outside of the scope of the Darkstar project is sufficient.

Edit: In addition, I still prefer an opt-in serialization model a la Hibernate or WCF and I believe that Darkstar would be vastly superior were it to include this, unless I missed something where it already does?

Quote:Original post by arbitus

Quote:Original post by jeffpk

All MMOs I know of (and I've worked directly with a number of groups building them over the years) keep their primary game state datastore in non-transactional in-memory systems.

Enterprise developers, by contrast, do everything in durable ATOMIC transactions.

These are not entirely true statements. MMOs are more likely to use a non-transactional model and enterprise developers are more likely to use a transactional model, but this is more of an implementation detail.

But the implementation is where the bugs, limitations and issues arise. No?

Quote:MMOs will not use databases for game state management, but will use databases for game information collection. The in-memory model will offloaded to a data store at intervals and save off for use in data-mining and reporting.

Fair enough. A PDS app can do the same thing if it wants.

Edit:
The PDS data store is not intended to replace an SQl database. It replaces the in-memory data model. In doing so it makes it more flexible, more robust and much more scalable then an in-memory system can.

Is *that* maybe the cause of confusion here? I was hoping my previous post on the goals made that clear. I apologize if it didn't. Its not a simple system to understand the under-pinnings of. It took about 5 years and 3 prototypes just to develop the architecture. Its a bit of a paradigm shift and those are always tricky to explain.

Quote:
This is the source of much of the criticism of the backing store of Darkstar and it is valid. However, the statements that you made are true: Darkstar's goal is not to be the most robust, industry-breaking MMO back-end.

Well, its goal is not to be a game engine.

Its goal IS to make the implementation of those game engines easier, more scalable, more flexible and more reliable then they are today.

Call me grandiose, but that was the vision and I think we're well on our way to proving it out. *shrug*

Quote:
It is meant to be a solid solution for scalable online projects, targeted mostly at hobbyists.

No. Completely and totally wrong. I just finished a project building an online game system for a 2.5 million dollar, venture capital funded company with PDS (among other technologies.) That's the one I will be lecturing on at AGDC next week.

I am starting a 4 million dollar PDS project for a major already successful successful game company right now.

Quote:
But you don't have to attack the credibility and the expertise of those who are questioning the applicability of Darkstar in certain situations. Acknowledging that the functionality is outside of the scope of the Darkstar project is sufficient.

What is "out of scope." Im happy to admit that all game engine functionality is out of scope. PDS is a platform for game engines, not a game engine itself. There are many game functions it does not provide. But it does provide the environment to build them and the hooks to tie them in.

But it is *definitely* intended as a piece of enterprise grade software for the enterprise of building MMOs. Being led and primarily implemented by a company with a history of producing enterprise grade products. (Solaris, Glassfish... etc)

In general, all new technology is adopted first by those with the least to risk. This is normal and no different from the way industry standards such as Renderware or Havok got their start. So. on a marketing level, do I ex[ect pick up first from indy's and hobbiests? Sure, that's the nature of the business. To be honest, PDS has actually had better initial pick up by big successful game companies then I told Sun to expect.

[Edited by - jeffpk on September 12, 2009 9:59:26 PM]

Quote:
Edit: In addition, I still prefer an opt-in serialization model a la Hibernate or WCF and I believe that Darkstar would be vastly superior were it to include this, unless I missed something where it already does?

GREAT question :)

You don't have to use the in-built persistence long term if you don't want to.

The Monkey Wrench, Rebel Monkey's platform, ran two different kinds of applications.

Our multi-player "mini-games" got all their data on the user passed in from the MySql database and wrote all game results back out to it. This was because the
rest of our social environment was web based and needed that data.

On the other hand, our "world" had long term persistence of its state and we let the PDS data store take care of that for us at zero work cost to us.

The DataStore is one of many pluggable interfaces and you can replace it if you want with anything that will meet the performance and transactional needs of the system. In point of fact the team has provided a pure in-memory data store for experimental purposes.

HOWEVER multi-node load balancing and fail-over is predicated on a shared data store between the nodes. So if you want those features, your DataStore has to implement that sharing.

Quote:
But the implementation is where the bugs, limitations and issues arise. No?

Of course, but the fact is that the model does not drive the implementation, meaning that you cannot claim it is the fault of the non-transactional model. There are many games that do not suffer from the same drawbacks.

Quote:
Fair enough. A PDS app can do the same thing if it wants.

I agree and acknowledge this, however, I would find PDS less attractive if I needed to bootstrap my own data store and persistency implementation on top of it. This is just a personal opinion however.

Quote:
Well, its goal is not to be a game engine.

Agreed, however many of the claims made about PDS are game related and I believe are a bit misleading. For instance, the claims about zoning and sharding are a bit disengenuous because of the general use of zoning and sharding as game-play mechanics rather than physical limitations.

Quote:
Its goal IS to make the implementation of those game engines easier, more scalable and more reliable then they are today.

Call me grandiose, but that was the vision and I think we're well on our way to proving it out. *shrug*

And I certainly wish you luck in this endeavor and will watch the progress of PDS. As I said before, I am only critical of the manner in which its public face has been presented in this thread.

Quote:
No. Completely and totally wrong. I just finished a project building an online game system for a 2.5 million dollar, venture capital funded company with PDS (among other technologies.) That's the one I will be lecturing on at AGDC next week.

I am starting a 4 million dollar PDS project for a major already successful successful game company right now.

Actually, you are quite correct, it was a very, very poor choice of words on my part and I apologize profusely. A more proper thing to say is probably that it is targeted at those with little to no experience in scalable server-side persistency and state management solutions. I understand it definitely has enterprise level application (why else would Sun pick it up?) and the word hobbyist has the wrong connotations.

Quote:
What is "out of scope." Im happy to admit that all game engine functionality is out of scope. PDS is a platform for game engines, not a game engine itself. There are many game functions it does not provide. But it does provide the environment to build them and the hooks to tie them in.

The data collection/mining part of the back-end that I mentioned in my previous post. And it is fine that it is not part of the PDS persistency, but you cannot pretend that the PDS model makes it unnecessary.

The issue mainly comes down to the fact that there is a lot of personal opinion that goes into how a particular group or individual would want to implement certain portions of the model that PDS is offering. All I have suggested in this thread is my own opinion, and I do not expect PDS to work the way I want it to. In fact, I admitted as much up front. But, what I don't like is the manner in which these opinions and suggestions are replied to as if they somehow are made irrelevant simply because PDS is a different implementation. The claims of bias against HPlus, the claims that those who support a queryable data-store format are "enterprise developers," on top of the broad claims of "problems" suffered by current MMOs all tend to undermine the actual positive attributes of PDS.

Some of us are real game developers with real game development experience, including MMOs. Some of us see real drawbacks in the PDS implementation and are expressing our opinions on the subject. That was the crux of my reply this afternoon, as it seems that you feel our comments and opinions are misguided or inexperienced. They aren't, they simply speak to the idea that PDS may not be a silver bullet for everyone. I believe you would admit as much anyway, so don't feel you need to downplay or even respond to every comment or suggestion someone puts forth, especially if you are going to do so by making assumptions about their skill or knowledge.

Quote:Original post by jeffpk

Quote:
Edit: In addition, I still prefer an opt-in serialization model a la Hibernate or WCF and I believe that Darkstar would be vastly superior were it to include this, unless I missed something where it already does?

GREAT question :)

You don't have to use the in-built persistence long term if you don't want to.

The Monkey Wrench, Rebel Monkey's platform, ran two different kinds of applications.

Our multi-player "mini-games" got all their data on the user passed in from the MySql database and wrote all game results back out to it. This was because the
rest of our social environment was web based and needed that data.

On the other hand, our "world" had long term persistence of its state and we let the PDS data store take care of that for us at zero work cost to us.

The DataStore is one of many pluggable interfaces and you can replace it if you want with anything that will meet the performance and transactional needs of the system. In point of fact the team has provided a pure in-memory data store for experimental purposes.

HOWEVER multi-node load balancing and fail-over is predicated on a shared data store between the nodes. So if you want those features, your DataStore has to implement that sharing.

How pluggable is the DataStore module as far as swapping storage formats without scrapping the module interface? For instance, if I wanted to change database providers, have some crazy idea about the blazing speed of XML, or want to do remoting?