Sign in to follow this  
Mantees

Managing large amounts of data

Recommended Posts

I am starting the development of a grid, turn based, strategic game, and I am evaluating the possibility to base it on very large maps. With very large maps, I mean maps that are so large that all the data cannot easily stay in RAM. I am wondering at what might be the best way to approach the problem (and if it is worth to :) ). One possible solution, with a relatively easy implementation, could be to use a DBMS like SQLite to base my data on. The DBMS would manage the caching depending on the available memory, which would make my job easier. An additional advantage would be that using SQL to access the data is easier and faster than writing my own optimized routines. But on the other hand I might get some other kind of performance problems. For sure with a proper designed database, accessing the data should be not so slow, but still it wouldn't beat memory stored linked lists. Is there a common solution for this kind of implementation? How does other data intensive games like Football Manager deal with the problem for example? Does my idea of a SQLite database makes some sense? Thanks

Share this post


Link to post
Share on other sites
Not fit in RAM? Just how big are we talking here?

I find it hard to believe you could possibly exceed a gigabyte of RAM (which you are guaranteed nowadays when you take swap into consideration), even if you load the entire game's map into memory: but working off the assumption that you are exceeding available memory, here's my advice:

A dedicated database system like [my]SQL[lite] would certainly make caching easier, but it may not be exactly suited to your needs - I doubt the SQL itself will be in anyway helpful if you're simply storing a grid map. Personally, I would write a quick implementation in SQL and check the performance - if it's acceptable, great, otherwise, a custom loading system may be required.

Share this post


Link to post
Share on other sites
In a (not so big) 10k X 10K tiles grid I have 100M tiles. Two bytes of information per tile makes 200Mb. If I have multiple layers (buildings, elevation model, natural resources, etc) for the same grid, it is 200Mb * layer, which can easily surpass 1Gb.

And of course I have also other data to store.

But I understand your point of a DBMS not exactly fitting my needs. I might need to go for smaller maps for now, and implementing a loading system later.

Share this post


Link to post
Share on other sites
Wow, that is big. O_O

Sounds like you're loading the entire game world at once... have fun :)

Share this post


Link to post
Share on other sites
Before you look into databases or the like, perhaps you should minimize the amount of tiles needed in memory. Not all tiles are needed at any given moment, since units will (hopefully) be scattered more or less in groups across the map. Quadtrees or KD-trees together with a streaming system can used to only keep needed tiles and their neighbors (how many neighbors depends on how fast any unit can possibly move) in memory.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this