Jump to content

  • Log In with Google      Sign In   
  • Create Account

We're offering banner ads on our site from just $5!

1. Details HERE. 2. GDNet+ Subscriptions HERE. 3. Ad upload HERE.


Databases in your games


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
19 replies to this topic

#1 MaxieQ   Members   -  Reputation: 128

Like
0Likes
Like

Posted 26 November 2012 - 10:08 AM

I have a hobby RPG project going called Endtime. My aim is to match the technical, game-play and artistic level of things like Fallout 2, Ultima, etc. Ten year old games. I think that's a feasible ambition for a solitary coder.


However, I won't really get away from a lot of data-shuffling for things like quests and items and such, and I was considering employing a database for easier back-end quest writing and such.

I mean, if I end up having more than 100 quests in my game, it is probably going to be easier to fill out the quest data in a form in a database than it would be to hard-code it in. I could store bulk quest data, dialog trees, the full item list in a database and call those when needed.


So, my question is simply because I am a bit fuzzy about how it would be done, do you use databases in your games and would it be worth trying to think up a bridge? Do you use databases in your games at all? If you do, do you use SQL or non-SQL ones?

Or if not, is it a really bad idea to use databases from an efficiency standpoint? Are they just too slow for things?
---
"Why do you knuckle-draggers insist on doing things the hard way... very well. " - Mr Burke

Sponsor:

#2 muratsal   Members   -  Reputation: 141

Like
0Likes
Like

Posted 26 November 2012 - 10:46 AM

i am using mongodb but join not enable and need to learn map-reduce :(
my map aplications http://haritaaraci.com

#3 ApochPiQ   Moderators   -  Reputation: 16079

Like
8Likes
Like

Posted 26 November 2012 - 11:14 AM

Databases are not the be-all and end-all of storing information.

I'd suggest looking into building a simple tool set that stores your data in a custom format or leverages an existing tool like XML/JSON/YAML/etc. You can knock together a tool to edit your game content in C# in a few hours; in the same time it'd take to integrate a database, design a schema, and start filling out "forms in a database", you could have some custom solutions that do precisely what you need and can be extended as you see fit whenever you like.

For perspective, most major game engines use custom data storage systems for precisely these reasons.

#4 Serapth   Crossbones+   -  Reputation: 5593

Like
1Likes
Like

Posted 26 November 2012 - 11:29 AM

Unless you are running massive numbers, most database systems are massive overkill for what you need. That said, they do solve a lot of problems that you might encounter ( transactions, concurrent request/race situations ) and theoretically are well optimized. If your data fits nicely to the database format it may certainly be a good way to go.

Consider a lightweight embeddable database like Firebird. It has ADO.net and ODBC connectors, making it fairly easy to access. ApochPiQ is right though, in many cases a database isn't really the best solution. In this case, embedding your data as code is a good fit. The above mentioned JSON, as well as LUA are both quite capable in this regard.

#5 ByteTroll   Crossbones+   -  Reputation: 1471

Like
1Likes
Like

Posted 26 November 2012 - 11:43 AM

Databases are not the be-all and end-all of storing information.

I'd suggest looking into building a simple tool set that stores your data in a custom format or leverages an existing tool like XML/JSON/YAML/etc. You can knock together a tool to edit your game content in C# in a few hours; in the same time it'd take to integrate a database, design a schema, and start filling out "forms in a database", you could have some custom solutions that do precisely what you need and can be extended as you see fit whenever you like.

For perspective, most major game engines use custom data storage systems for precisely these reasons.


I agree with ApochPiQ. I worked on a game where we used MySQL databases to manage everything -- quests, stats, players, items, etc. If I had to go back and do it again, I would have just wrote a custom storage format. The amount of trouble that those databases caused was unbelievable; especially when you have more than one programmer working on the game.
▬▬▬▬▬▬▬▬▬▬▬ஜ۩۞۩ஜ▬▬▬▬▬▬▬▬▬▬▬▬▬
I see the future in 1's and 0's
▬▬▬▬▬▬▬▬▬▬▬ஜ۩۞۩ஜ▬▬▬▬▬▬▬▬▬▬▬▬▬

"This is called programming. The art of typing shit into an editor/IDE is not programming, it's basically data entry. The part that makes a programmer a programmer is their problem solving skills." - Serapth

#6 MaxieQ   Members   -  Reputation: 128

Like
0Likes
Like

Posted 26 November 2012 - 11:49 AM

Thanks guys. I really appreciate your comments. I'll have a look at XML or JSON if I can get that to work with Unity Free, which is my engine of choice.
---
"Why do you knuckle-draggers insist on doing things the hard way... very well. " - Mr Burke

#7 Steve_Segreto   Crossbones+   -  Reputation: 1541

Like
2Likes
Like

Posted 26 November 2012 - 11:52 AM

In my opinion, databases are meant to solve a different problem than what you describe. They are designed for (hundreds) thousands (millions) of users to insert/remove/query the information in them concurrently. Not for a single programmer to store inter-related program data. As others have pointed out, if you use a professional database solution, you will add complexity to your own life with very little return on investment.

#8 Serapth   Crossbones+   -  Reputation: 5593

Like
1Likes
Like

Posted 26 November 2012 - 12:44 PM

In my opinion, databases are meant to solve a different problem than what you describe. They are designed for (hundreds) thousands (millions) of users to insert/remove/query the information in them concurrently. Not for a single programmer to store inter-related program data. As others have pointed out, if you use a professional database solution, you will add complexity to your own life with very little return on investment.


Oh, there is a massive return on investment, especially if you are creating say.... an MMO.

But it all comes down to weighing the upsides vs the downsides, and the reality is, most games dont use a portion of the ability of most databases on the market.

That said, MySQL is a notorious pain in the ass, so that would be a big part of it. There are many databases that were designed specifically to be embedded in an application and they will give a much different experience.

#9 MaxieQ   Members   -  Reputation: 128

Like
0Likes
Like

Posted 26 November 2012 - 01:20 PM

I'm an on- and off player of Eve Online, and my thinking about databases in games started because I know that Eve invests a lot in their database. Of course, Eve is an MMO with hundreds of thousands of players, so there is the question of scale. But it got me started thinking about the utility of databases in games.

Again thanks everyone for your comments. It's most instructive.
---
"Why do you knuckle-draggers insist on doing things the hard way... very well. " - Mr Burke

#10 ApochPiQ   Moderators   -  Reputation: 16079

Like
1Likes
Like

Posted 26 November 2012 - 03:57 PM

Even traditional relational (SQL-oriented) databases are of questionable utility to a modern MMO. Reliable distributed data storage is often more valuable than strict ACID compliance and the other features of relational DBs, at least until you get to the archival/long-term-retention state of affairs.

#11 ~Helgon   Members   -  Reputation: 357

Like
1Likes
Like

Posted 26 November 2012 - 05:21 PM

I'm not sure, but even a big mmo like Aion is using xml to save data- can't be that wrong

from time to time i find time


#12 Michael Tanczos   Senior Staff   -  Reputation: 5437

Like
0Likes
Like

Posted 26 November 2012 - 06:43 PM

http://www.sqlite.org/

Writing a storage solution is not going to be quicker than it would take to implement a mature database product like sqlite. Something like this product gets you focusing on what you want to store rather than how you are going to store it. Oh, and the license is "public domain".

There are a vast number of existing tools out there that are already available to work with existing database products. You don't have to create any tools for this.

#13 ApochPiQ   Moderators   -  Reputation: 16079

Like
1Likes
Like

Posted 26 November 2012 - 06:53 PM

Having seen SQLite used for game development in the past, I'm actually more against using that than using more traditional DB systems. It claims to be ACID compliant but it's easy to break that aspect if you implement it wrong, and treating it like a genuine database is begging for weird bugs.

I stand by my original statement: by the time you integrate even something like SQLite, get up to speed on the tools, etc. you can hammer out a fully custom solution that beats the pants off a generic database system. Existing libraries for storage formats are so prevalent that there's no call for rolling your own storage solution. You can save all the effort of integration and convert it into effort making your ecosystem better, and that pays off a lot more over time in my experience.

#14 Michael Tanczos   Senior Staff   -  Reputation: 5437

Like
0Likes
Like

Posted 26 November 2012 - 08:46 PM

Having seen SQLite used for game development in the past, I'm actually more against using that than using more traditional DB systems. It claims to be ACID compliant but it's easy to break that aspect if you implement it wrong, and treating it like a genuine database is begging for weird bugs.


This is just my curiosity here but what situation did you see SQLite used that it didn't achieve what it was designed to do? Was the problem just programmer error?

I was just thinking for his situation he's looking at storing a limited amount of data without much need for high end database features.

#15 3Ddreamer   Crossbones+   -  Reputation: 3159

Like
0Likes
Like

Posted 27 November 2012 - 01:45 AM

MaxieQ,

Game development office database and online game server database are things to research and eventually launch. As your organization grows, the need for these two databases will surely increase.


Game source code modulization might be better served by avoiding a database integration in the game source code. It is quite easy for data in a database to get lost or detached from the game source code. If changes in game code will require a change of database structure, then you just doubled your work and multiplied your risk of a broken game.

On the other hand, maintaining a database can make changes to data itself quick and easy if the coding is safe and reliable. Tree and kinds of data hierarchy would be a must for complex games in general if you are going to use a database.

I am sure that fans of game databases will tell you that they are glad they learned and integrated them. On the other hand, with games slowly getting more complex by the day, the need for dynamic writing with modulization seems to me to work against a database.


Clinton

Personal life and your private thoughts always effect your career. Research is the intellectual backbone of game development and the first order. Version Control is crucial for full management of applications and software.  The better the workflow pipeline, then the greater the potential output for a quality game.  Completing projects is the last but finest order.

 

by Clinton, 3Ddreamer


#16 Wilhelm van Huyssteen   Members   -  Reputation: 1002

Like
0Likes
Like

Posted 27 November 2012 - 02:10 AM

I personally cant imagine an MMO-scale project not using a traditional database... But then again im a profesional plsql developer who spends his work day writing tedious finance and stock related batch jobs or designing and implementing new bussiness applications starting wit the database and ending with the actual application so I might be biased... In my personal project http://www.gamedev.net/blog/blog-807/cat-252-strife I use an oracle database and for me it fits in perfectly. Although. I dont store any quests or items in the database, I have a custom format for that since it makes more sense to me. The database is instead is used to store things like accounts, characters and links from a character to all its items and quests. Meaning it only stores id's for quests and items in the database but the actual items and quests are stored elsewhere since It doesn't seem benificial to store static data in the database (items and quest doesnt change while the server is running).

EDIT: Another reason why you dont want to store static quest and item information on your server side database is that the client also needs access to it... If a player wants to read a quest the server doesnt (or at elast shouldnt) send over the complete quest dialogue and everything else about about the quest. Instead it should only send over a quest id and then the client should look it up from its own data)

Edited by Wilhelm van Huyssteen, 27 November 2012 - 02:40 AM.


#17 Serapth   Crossbones+   -  Reputation: 5593

Like
0Likes
Like

Posted 27 November 2012 - 08:51 AM

Even traditional relational (SQL-oriented) databases are of questionable utility to a modern MMO. Reliable distributed data storage is often more valuable than strict ACID compliance and the other features of relational DBs, at least until you get to the archival/long-term-retention state of affairs.



Having played around with NoSQL databases recently ( CouchDB ), I can't imagine they are the answer! Maybe my brain is too wired to think in terms of tables, but my god the experience sucked beyond simple document storage. If you are storing your data as a blob ( say a chunk of JSON ) that you grab, update then push back to the database, it works smashingly, but updating a single value within that document causes an entire new revision to be created. Now, storing files in a NoSQL database is an order of magnitude easier then a relational database, but actual data, not so much.

#18 Michael Tanczos   Senior Staff   -  Reputation: 5437

Like
0Likes
Like

Posted 27 November 2012 - 09:21 AM

Having played around with NoSQL databases recently ( CouchDB ), I can't imagine they are the answer! Maybe my brain is too wired to think in terms of tables, but my god the experience sucked beyond simple document storage. If you are storing your data as a blob ( say a chunk of JSON ) that you grab, update then push back to the database, it works smashingly, but updating a single value within that document causes an entire new revision to be created. Now, storing files in a NoSQL database is an order of magnitude easier then a relational database, but actual data, not so much.


Well.. that's one example of a NoSQL database. Compare that to Redis (http://www.redis.io) which is used by Blizzard Entertainment, Stackoverflow, Github, Flickr, Craigslist.. we use Redis for custom news views on the frontpage as well as for real-time access to member data in the Top Members section (http://www.gamedev.net/sm/)

Blizzard uses an 8 node redis cluster for serving avatars in World of Warcraft. Redis is an in-memory data structure server with persistence capabilities.

Also with your situation games not necessarily save in realtime to disk.. with most it's sufficient to do periodic checkpoint saves.

Edited by Michael Tanczos, 27 November 2012 - 09:23 AM.


#19 Serapth   Crossbones+   -  Reputation: 5593

Like
0Likes
Like

Posted 27 November 2012 - 11:10 AM


Having played around with NoSQL databases recently ( CouchDB ), I can't imagine they are the answer! Maybe my brain is too wired to think in terms of tables, but my god the experience sucked beyond simple document storage. If you are storing your data as a blob ( say a chunk of JSON ) that you grab, update then push back to the database, it works smashingly, but updating a single value within that document causes an entire new revision to be created. Now, storing files in a NoSQL database is an order of magnitude easier then a relational database, but actual data, not so much.


Well.. that's one example of a NoSQL database. Compare that to Redis (http://www.redis.io) which is used by Blizzard Entertainment, Stackoverflow, Github, Flickr, Craigslist.. we use Redis for custom news views on the frontpage as well as for real-time access to member data in the Top Members section (http://www.gamedev.net/sm/)

Blizzard uses an 8 node redis cluster for serving avatars in World of Warcraft. Redis is an in-memory data structure server with persistence capabilities.

Also with your situation games not necessarily save in realtime to disk.. with most it's sufficient to do periodic checkpoint saves.


I actually wanted to use Redis, it seemed like the much more sane option than CouchDB. There is a gigantic problem I ran into here.

Redis isn't on Windows yet, at least not in any official capacity. It has been in a "coming soon" state for far too long. What I heard about stability of the Windows port was... not good.

#20 Michael Tanczos   Senior Staff   -  Reputation: 5437

Like
0Likes
Like

Posted 27 November 2012 - 11:25 AM

Redis isn't on Windows yet, at least not in any official capacity. It has been in a "coming soon" state for far too long. What I heard about stability of the Windows port was... not good.


Yeah Redis isn't one of those solutions you package with a game for distribution. You'd need to be running servers users will connect to. It's lean enough though that on any LAN you could always throw an old box up with linux and run it for testing purposes (or run linux inside a VM with Redis on it).




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS