Quote:Original post by Sirisian
During midterms people are printing a ton not to mention there are real-time applications that query the database constantly for data

Just a little nitpick. Real-time doesn't mean really really fast. It means that system guarantees a deadline by which a task will complete.

For example, consider sampling of notches on CD. The system needs to read data at exactly given rate (x GHz). If it's a bit late, or a bit early, the data will appear corrupt. Trivial system, one event, one period, yet it needs real-time guarantee. I remember problems just like that with old serial ports, where, if you couldn't process the signal on time, it was lost, and likely, incoming data became unreadable.

If these databases were soft real-time, then as soon as the system would become overloaded, it would start dropping queries immediately. This is a major challenge in such systems, and MMO design. What to do once you exceed the available resources. Trivial solution is simply to crash the server, but any practical solution will need to account for this overload, and either re-balance resources, decrease fidelity, or release part of load (disconnect clients or something) - all while not violating the real-time constraint.

The fact that scalable systems exists is just that - a fact. But few such systems have any kind of real-time demands. A 2.4 second delay while spooling a document isn't relevant or noticable.

Quote:Original post by hplus0603

Quote:. As long as you're not updating all the players at the same time it's probably not that big of a deal.

Not updating all players at the same time can lead to the following problem:
** Snip **

No I meant don't queue up all the changes and thrash the server. If you do a trade make the transaction right there and then. Don't wait and perform a huge update all at once.

I've noticed that databases seem really fast because the transactions for most programs are very infrequent even though there might be a lot of them so you're never handling say 10000 transactions at the same time. How many times do things like trades and transactions happen? But you know someone out there is going to try to iterate all the players and run the database persistence in one go. I was just pointing out that you shouldn't do it like that. Unless I'm mistaken and that is the right way? (I use flat files at the moment for my stuff).

Quote:Original post by Sirisian

Unless I'm mistaken and that is the right way? (I use flat files at the moment for my stuff).

Same problem with transactions remain. How do you ensure integrity of writing to files?

Consider the above trade case - you write first part of transaction, then something happens (out of disk space, server crash, etc.). Or half-way during this write, another part of application reads one of player's data, obtaining partial and logically invalid state.

And that's just the simple, possibly single-threaded model. Except that these situations are rare enough to go unnoticed, but only start showing up as unexplained and unnoticed errors later on as load increases.

Quote:Original post by hplus0603

Quote:How do you ensure integrity of writing to files?

You can use fsync() and journaling to implement transactions in files.
First, have a header which points at the actual data.
Second, don't overwrite the old data. Instead, append the new data at the end.
Third, flush the new data, so you know it's in the file.
Fourth, re-write the header to point at the new data instead of the old data.
Fifth, flush the file header -- this is the "commit" step.

If you crash anytime before Fifth, you will end up with a file containing the old data.

I mentioned this because I once had to deal with this very problem with a 100+ page Word Perfect document.

I believe it was one of first WYSIWG versions, and the working copy was on floppy. During save, there wasn't enough space. Although error was reported, it was no longer possible to save the file elsewhere, resulting in an error message. Apparently, part of file was stored on disk, and the initial failure to save corrupted the working copy.

File was comprised of descriptor header, and actual data block. One of members of header was length of data block.

The usual algorithm for updating such headers where total length of data is unknown before it's actually written is to set header length to 0, then use it as counter while serializing the data.

But for a reason unknown to me, the value of 0 was actually written on the disk first. When writing the data block there wasn't enough space, so the rest of the process failed, leaving file corrupted.

After figuring out the data format, the fix was trivial - manually deducing the length and setting that repaired the format.

I still find it surprising though, how many similar types of errors exist in many applications, since such corner cases are rare enough to pass as good enough. It also demonstrates a fairly common scenario (disk space) that can cause a fairly complex error situation when dealing with distributed resources (part monolithic data, part stored on disk, part determined during run-time).

(oh I didn't mean anything by the flat file comment. It's just a hack really quick while I test things. It has no transaction system).

Sweet idea hplus. I have some ideas for some data structures. Allocate a 0.5 MB file for each player (or one huge file with all the characters). Then from the top just create the headers that point to objects and expand those to point to other objects. Loading the data structure into memory would quickly allow it to be defragmented. Also using your idea to allow it to not lose data upon a crash. Sounds like a flat file database (I've never actually made one of those) except the Wikipedia article says "There are no structural relationships" and the one I just explained has those (definitely more along the lines of an object DB). Sounds very interesting. I think I'll try to make something like what I explained later when I have the time :)

Quote:Original post by hplus0603

Quote:You shouldn't design your MMO that it has to be doing DB reads every time you move around etc.

Which is why the design of the front-end (by which I mean front-end to the database -- i e, middle tier, or "game server") matters.

However, I have to ask you: have you shipped and supported a MMO game? The Cryptic guys have, and they had reasonable justification for going away from RDBMS.

I thought some of their justifications looked like "we understand our game far better than we understand RDBMSs". It's sort of admirable if you have a certain design and can't compromise on it in order to meet persistence requirements, eg. wanting to use the bandwidth associated with thousands of players while keeping the minimal latency you'd see in a single player game. But I think 95% of people could simply alter their game slightly to accommodate the way the DB works.

We save stuff continuously and asynchronously while a player is logged in and performing actions, and performance so far is fine. The code is a bit cumbersome in places, but I guess you win some, you lose some.

Quote:Original post by hplus0603If you really know what you're doing, picking an RDBMS over something like an ODBMS is not as clear-cut as you claim. And you don't have to invent your own; there are plenty to choose from in the market.

In my mind, the question comes down to: do you need the ad-hoc query capability of your data? We do, so we stick with an RDBMS, but it does have a price (dollar cost).

I don't necessarily say ODBMS is worse, but I think the fact there are a lot of people out there who are very skilled with traditional DBs makes using another option more risky... if you are going to get a proper DB guy and not let your programmer do the DB of course. If I was doing a big MMO I would want a specialist DB guy on board, and I imagine it's easier to find such a guy for a standard relational DB. To me, using something else would be more risky and probably incur extra cost up front investigating it... even if it might reduce hardware costs once the number of users starts to ramp up.