5 replies to this topic

[Xanather]

I was wondering if any of these ideas could even be possible on hardware today:
Quick overview: A database that will hold probably 1 million+ accounts (probably a overkill in the amount of numbers, but id like to be safe and try to design a system that is easily expandable).
The database will probably all contain, for example, the following...
a) personal information (username, password aswell)
b) list of characters on the account...
- i) character level
- ii) character inventory
- iii) more game-specific data

Thoughts on creating this database:
1. Design my own database system that has a folder containing the account database. There will be 1 .dat file per account. Ive tried something similar like this before and the folder size becomes really large even though there are not large files containing within it, this probably has something to do with the disk state and NTFS... Although I dont think performance would be affected as the database size increases if this is the case though (directly load the single file, no searching required).
2. Same design as above but instead a maximum of 200 accounts per file (not completely sure how this will work though, by that I mean how to find the correct account between multiple files).
3. One main file, dynamically add/remove accounts from this file (I would assume though searching for a account in this solution will have the same problems that SQL databases yield).
4. Just go with technologies such as SQL (big databases become slow though).

The question if i did not clarify: Any of the suggested ideas above even feasibly possible?

As always, thank you. All replies are appriciated.
Xanather.

Edited by Xanather, 27 July 2012 - 03:21 AM.

[hplus0603]

A single database table with a hundred million records is not a problem. Storage is cheap and easy to manage.

The question is how many queries per second you want to run on that table. With proper SSDs and a 10:1 read/write distribution, and using read slaves for the read queries, I'd expect you could do 1,000 writes and 10,000 reads per second to a SQL database like that. Note that read slaves are very easy to set up for SQL databases, but much harder to set up for file system based solutions.

You could also put all the accounts in a single Redis instance. I bet you could do several hundred thousand transactions per second there, with the trade-off that if the system crashed, you might roll back to the last disk snapshot (20 minutes or so for a full system.) Redis also supports replicas that you can use as read slaves.

So, to better answer your question about feasibility: The number of accounts doesn't matter much. The number and complexity of queries per second is much more important. Do you have a feeling for what you'd want to accomplish there?

Edited by hplus0603, 27 July 2012 - 03:57 AM.

[Ashaman73]

1. Design my own database system that has a folder containing the account database. There will be 1 .dat file per account. Ive tried something similar like this before and the folder size becomes really large even though there are not large files containing within it, this probably has something to do with the disk state and NTFS... Although I dont think performance would be affected as the database size increases if this is the case though (directly load the single file, no searching required).
2. Same design as above but instead a maximum of 200 accounts per file (not completely sure how this will work though, by that I mean how to find the correct account between multiple files).
3. One main file, dynamically add/remove accounts from this file (I would assume though searching for a account in this solution will have the same problems that SQL databases yield).
4. Just go with technologies such as SQL (big databases become slow though).

1,2,3 : do this only if you want to write your own DB for fun, not if you need a scalable, working, fast data base.
4: this is the way to go, 1 mio. entiries and a handful of tables are not really a problem. With the right indicies you will have fast access times, the question is, how often will you access the db as hpplus already explained.

From the data you describes access will be most likely not happen very often (login,logout, 5 minut autosave), in this case a simple mysql data base will be likely sufficient and can be scaled up later once you need to support more accounts.

[Xanather]

Id probably predict... hmm... 2-4? queries per second if I design it well (may probably increase much more though, variates greatly depending on current amount of players obviously). Does this sound feasible for SQL?

Edited by Xanather, 27 July 2012 - 08:43 AM.

[Firestryke31]

Here's a hint: This site uses a SQL database. It has 4,362,189 Total Posts with 1857 users online (including guests).

It has to sort through all of those posts every time anyone views any thread. Post edits and new posts are written to the database pretty often. Does the site seem responsive? It does to me. I think a SQL database can handle a quarter of that with only a few queries a second just fine.

In this day and age, the only reason to write database software yourself is as a learning experience. If you're serious about performance, stick with the proven projects dedicated soley to building a fast database engine. Hell, MySQL's pretty damn fast and it's free.

[Xanather]

Haha you are completely right. Thanks