dealing with terrain chunks for a big world

the game in question: AIRSHIPS!, a WW1 era alternate history airship flight sim...

http://www.gamedev.net/gallery/album/672-rockland-softwares-airships/

the game world is very large, stretching from the arctic circle to north africa, and from the mississippi to the urals.

the plan is to create or generate a heightmap which can be used to generate ground mesh chunks.

the game uses a scale of 1 foot = 1d3d unit.

world coordinates consist of sector x,z (a sector is 10K feet across), and a location in that sector (x,y,z).

a mesh scale of one vertex per ten feet is planned. with a sector size of 10K feet, a 1000x1000 vertex mesh is a natural first choice for chunk size.

but here's the kicker: the world is 4912 x 1637 sectors in size. if a sector is a terrain chunk, that's 6,866,304 chunks! at a cool 1 million verts EACH!

so... storing it all in ram is out (wasn't even thinking about it), storing the heightmap in ram is probably no go as well. using one byte for height values, it would be 6,866 gig in size. so maybe i need a lower resolution (coarser) heightmap to cut down on the heightmap size, and add in some "heightmap splatting" to provide high rez detail? or maybe go procedural for the heightmap, then generate chunks on the fly as needed and save to disk once generated?

the whole thing is doable, you procedurally (or hand) generate the heightmap (at some resolution), then use it to generate ground meshes. its just the size of all the data and initial generation time involved that makes implementation something to think about. and do you do it before release and include just the meshes? or the meshes and the heightmap? or do you do it at install? so you don't have to redistribute all that content? actually, based on the amount of data involved, generating meshes on the fly seems the only way, then save them to disk for reuse next time instead of re-generating on the fly again - assuming on the fly is slow enough to warrant it. in caveman i can generate 3 300x300 chunks of up to 15,000 renderables each, plus 4 interleaved ground meshes per chunk which are generated on the fly, so fast there's not even time to display any kind of "loading area" message (<1/2 second).

in the past, i've gotten good results with meshes generated on the fly from procedural heightmaps.

Since you only need to compute the heightmap once, one way to do it is to build it to one huge file and then use streaming to read renderable parts. If you allocate enough buffers to contain maximum displayed range, you only need to remove chunks out of range (after some countdown timer maybe, to delay removal a short while) and read new displayable chunks into empty buffers. Then it's just a matter of displaying the list of buffers

You can utilise geometry shaders to create the terrain mesh dynamically from the heightmap buffers, and they can also be used for LOD stuff

Edit: Maybe zlib or something could be used to compress chunks in the heightmap to get it down from that 6,8GB?

Look at how flightgear does this.

It uses openscenegraph I believe, it's all open source so you can scrutinise it and see how it works. It streams the entire world in and out as you fly over it, in real scale, with millions of vertices and spanning about 15gb for the full tile set.

Good luck!

You can also only save a low level of detail version on disk, and generate the more detailed levels procedurally (perhaps based on a couple of parameters to determine how bumpy it should be etc)

Ability to determine where the line between stored and generated goes for each map chunk could be used to optimize (are all levels of detail stored, or maybe only the lowest resolution one?).

Of course you can also compress the stored heightmap regions using various methods which should help quite a bit.