I am actually implementing a voxel based terrain in our MMO (Abydos Online).

I wrote a simple C module, 1000 lines of code (500 for functionality and 500 unit/integration tests) that reads/write voxels to disk.

Here is how it works:
The terrain is called a volume.
A volume is divided into regions (default 8x8x8 regions).
One region is divided into blocks (default 16x616x16 blocks).
One block is 32x32x32 voxels (1 voxel = 1 cubic meter).
Each voxel is 4 bytes, 1 byte density, 2 byte material and 1 byte reserved.

When terrain is initiated, the C module creates a directory with one file per region. If the region is homogenous, only that file exists which contains the voxel for the whole region (for example all air, dirt, water etc). The filename would be "region_x_x_x.dat" where the x is replaced with the x, y and z position.

If a region is not homogenous, a directory "region_x_x_x" is created which contains the blocks, by default 16x16x16 blocks result in 4096 files, each of this file is named "block_x_y_z.dat". If this file will either be 1 voxel (4 bytes) in size of a homogenous block or 32x32x32 voxels (128 kb) for a mixed block.

A block with all the 128 kb of data is referred to as a chunk.

Currently working:
When a player logs in, the whole terrain is read and sent back to the player, regions and blocks are sent one as one TCP packet back to the client, chunks divided into parts of 8kb each and reassembled in client.
The client stores the data in the same structure as the server but also parses all regions/blocks/chunks and build static terrain geometry of this.

Todo:

• Terrain versioning, each region/block/chunk should have a version number (2bytes, 4 bytes, 8 bytes?) that is increased per change. At one point the max version will likely be reached and you will have edge cases where players that should download updated terrain does not. A straight forward solutions is to have a dirty flag forcing all players to re-download the whole terrain and restart versioning. (Other solutions probably exists, but I think this is the best one since it does does not have any edge cases)
• Each region/block should have a sort of level of detail (LOD).
• Server should have two caches, one for reading voxels and one for writing.
• Reading buffer - When a read request, check buffer first, if not in buffer read requests terrain and its nearby terrain data and store in buffer (memory).
• Writing buffer - When writes requests are done, don't actually write at once but wait for some time possible save up some writes and do them all at once.
• Server should propagate changes in the terrain to all clients (at the moment, clients have to restart to get updated terrain).
• Server/Client should compress chunks before storing on disk (bzip2/gzip).
• Writing requests to the C module should be replicated to other areas that "run" the same area, making it scale horizontally. This allows servers to come out of "sync" with their terrain. A sane voting protocol (probably using the versioning if possible) would have to be implemented to allow servers that has gone offline to sync up with the other servers.
• C module improvement could be done by some threading (I will try to avoid this as long as possible).

Not needed:
Clients does not need to request writes to the server side terrain since all logic run on the server. (For less "safer" solutions, this is just a matter of adding a new message type on the protocol). Write requests that are issued from the clients are possible more open for exploit.

Problems:
Server disk space not much of an issue, our plan is to have 10x10 areas of 8x8x8x -> 16x16x16 -> 32x32x32 voxels (256 GB of data per area in worst case). Which means 25 terra byte for the whole world.
Client side, this would of course be a problem, but worst case is highly unlikely to be achieved, most areas will likely have many homogenous parts, for example half the area would be air, other half dirt/rock and most of the mixed chunks would be the surface or mines. Some gameplay mechanics could be implemented that tries to make parts of the terrain homogenous under certain circumstances.
One could have a special storage cache rules for the client, for example not allowing storing more than X GB of data on disk, freeing up space as new data is written (size could be set by the players themselves but should have a minimun value for sanity). This also requires clients to request syncing a full area since the logic on the server would have this client information.

I am actually implementing a voxel based terrain in our MMO (Abydos Online).

I wrote a simple C module, 1000 lines of code (500 for functionality and 500 unit/integration tests) that reads/write voxels to disk.

Here is how it works:
The terrain is called a volume.
A volume is divided into regions (default 8x8x8 regions).
One region is divided into blocks (default 16x616x16 blocks).
One block is 32x32x32 voxels (1 voxel = 1 cubic meter).
Each voxel is 4 bytes, 1 byte density, 2 byte material and 1 byte reserved.

When terrain is initiated, the C module creates a directory with one file per region. If the region is homogenous, only that file exists which contains the voxel for the whole region (for example all air, dirt, water etc). The filename would be "region_x_x_x.dat" where the x is replaced with the x, y and z position.

If a region is not homogenous, a directory "region_x_x_x" is created which contains the blocks, by default 16x16x16 blocks result in 4096 files, each of this file is named "block_x_y_z.dat". If this file will either be 1 voxel (4 bytes) in size of a homogenous block or 32x32x32 voxels (128 kb) for a mixed block.

A block with all the 128 kb of data is referred to as a chunk.

Currently working:
When a player logs in, the whole terrain is read and sent back to the player, regions and blocks are sent one as one TCP packet back to the client, chunks divided into parts of 8kb each and reassembled in client.
The client stores the data in the same structure as the server but also parses all regions/blocks/chunks and build static terrain geometry of this.

Todo:

• Terrain versioning, each region/block/chunk should have a version number (2bytes, 4 bytes, 8 bytes?) that is increased per change. At one point the max version will likely be reached and you will have edge cases where players that should download updated terrain does not. A straight forward solutions is to have a dirty flag forcing all players to re-download the whole terrain and restart versioning. (Other solutions probably exists, but I think this is the best one since it does does not have any edge cases)
• Each region/block should have a sort of level of detail (LOD).
• Server should have two caches, one for reading voxels and one for writing.
• Reading buffer - When a read request, check buffer first, if not in buffer read requests terrain and its nearby terrain data and store in buffer (memory).
• Writing buffer - When writes requests are done, don't actually write at once but wait for some time possible save up some writes and do them all at once.
• Server should propagate changes in the terrain to all clients (at the moment, clients have to restart to get updated terrain).
• Server/Client should compress chunks before storing on disk (bzip2/gzip).
• Writing requests to the C module should be replicated to other areas that "run" the same area, making it scale horizontally. This allows servers to come out of "sync" with their terrain. A sane voting protocol (probably using the versioning if possible) would have to be implemented to allow servers that has gone offline to sync up with the other servers.
• C module improvement could be done by some threading (I will try to avoid this as long as possible).

Not needed:
Clients does not need to request writes to the server side terrain since all logic run on the server. (For less "safer" solutions, this is just a matter of adding a new message type on the protocol). Write requests that are issued from the clients are possible more open for exploit.

Problems:
Server disk space not much of an issue, our plan is to have 10x10 areas of 8x8x8x -> 16x16x16 -> 32x32x32 voxels (256 GB of data per area in worst case). Which means 25 terra byte for the whole world.
Client side, this would of course be a problem, but worst case is highly unlikely to be achieved, most areas will likely have many homogenous parts, for example half the area would be air, other half dirt/rock and most of the mixed chunks would be the surface or mines.
One could have a special storage cache rules for the client, for example not allowing storing more than X GB of data on disk, freeing up space as new data is written (size could be set by the players themselves but should have a minimun value for sanity). This also requires clients to request syncing a full area since the logic on the server would have this client information.

I am actually implementing a voxel based terrain in our MMO (Abydos Online).

I wrote a simple C module, 1000 lines of code (500 for functionality and 500 unit/integration tests) that reads/write voxels to disk.

Here is how it works:
The terrain is called a volume.
A volume is divided into regions (default 8x8x8 regions).
One region is divided into blocks (default 16x616x16 blocks).
One block is 32x32x32 voxels (1 voxel = 1 cubic meter).
Each voxel is 4 bytes, 1 byte density, 2 byte material and 1 byte reserved.

When terrain is initiated, the C module creates a directory with one file per region. If the region is homogenous, only that file exists which contains the voxel for the whole region (for example all air, dirt, water etc). The filename would be "region_x_x_x.dat" where the x is replaced with the x, y and z position.

If a region is not homogenous, a directory "region_x_x_x" is created which contains the blocks, by default 16x16x16 blocks result in 4096 files, each of this file is named "block_x_y_z.dat". If this file will either be 1 voxel (4 bytes) in size of a homogenous block or 32x32x32 voxels (128 kb) for a mixed block.

A block with all the 128 kb of data is referred to as a chunk.

Currently working:
When a player logs in, the whole terrain is read and sent back to the player, regions and blocks are sent one as one TCP packet back to the client, chunks divided into parts of 8kb each and reassembled in client.
The client stores the data in the same structure as the server but also parses all regions/blocks/chunks and build static terrain geometry of this.

Todo:

• Terrain versioning, each region/block/chunk should have a version number (2bytes, 4 bytes, 8 bytes?) that is increased per change. At one point the max version will likely be reached and you will have edge cases where players that should download updated terrain does not. A straight forward solutions is to have a dirty flag forcing all players to re-download the whole terrain and restart versioning. (Other solutions probably exists, but I think this is the best one since it does does not have any edge cases)
• Each region/block should have a sort of level of detail (LOD).
• Server should have two caches, one for reading voxels and one for writing.
• Reading buffer - When a read request, check buffer first, if not in buffer read requests terrain and its nearby terrain data and store in buffer (memory).
• Writing buffer - When writes requests are done, don't actually write at once but wait for some time possible save up some writes and do them all at once.
• Server should propagate changes in the terrain to all clients (at the moment, clients have to restart to get updated terrain).
• Server/Client should compress chunks before storing on disk (bzip2/gzip).
• Writing requests to the C module should be replicated to other areas that "run" the same area, making it scale horizontally. This allows servers to come out of "sync" with their terrain. A sane voting protocol (probably using the versioning if possible) would have to be implemented to allow servers that has gone offline to sync up with the other servers.
• C module improvement could be done by some threading (I will try to avoid this as long as possible).

Not needed:
Clients does not need to request writes to the server side terrain since all logic run on the server. (For less "safer" solutions, this is just a matter of adding a new message type on the protocol).

Problems:
Server disk space not much of an issue, our plan is to have 10x10 areas of 8x8x8x -> 16x16x16 -> 32x32x32 voxels (256 GB of data per area in worst case). Which means 25 terra byte for the whole world.
Client side, this would of course be a problem, but worst case is highly unlikely to be achieved, most areas will likely have many homogenous parts, for example half the area would be air, other half dirt/rock and most of the mixed chunks would be the surface or mines.
One could have a special storage cache rules for the client, for example not allowing storing more than X GB of data on disk, freeing up space as new data is written (size could be set by the players themselves but should have a minimun value for sanity). This also requires clients to request syncing a full area since the logic on the server would have this client information.

I am actually implementing a voxel based terrain in our MMO (Abydos Online).

I wrote a simple C module, 1000 lines of code (500 for functionality and 500 unit/integration tests) that reads/write voxels to disk.

Here is how it works:
The terrain is called a volume.
A volume is divided into regions (default 8x8x8 regions).
One region is divided into blocks (default 16x616x16 blocks).
One block is 32x32x32 voxels (1 voxel = 1 cubic meter).
Each voxel is 4 bytes, 1 byte density, 2 byte material and 1 byte reserved.

When terrain is initiated, the C module creates a directory with one file per region. If the region is homogenous, only that file exists which contains the voxel for the whole region (for example all air, dirt, water etc). The filename would be "region_x_x_x.dat" where the x is replaced with the x, y and z position.

If a region is not homogenous, a directory "region_x_x_x" is created which contains the blocks, by default 16x16x16 blocks result in 4096 files, each of this file is named "block_x_y_z.dat". If this file will either be 1 voxel (4 bytes) in size of a homogenous block or 32x32x32 voxels (128 kb) for a mixed block.

A block with all the 128 kb of data is referred to as a chunk.

Currently working:
When a player logs in, the whole terrain is read and sent back to the player, regions and blocks are sent one as one TCP packet back to the client, chunks divided into parts of 8kb each and reassembled in client.
The client stores the data in the same structure as the server but also parses all regions/blocks/chunks and build static terrain geometry of this.

Todo:

• Terrain versioning, each region/block/chunk should have a version number (2bytes, 4 bytes, 8 bytes?) that is increased per change. At one point the max version will likely be reached and you will have edge cases where players that should download updated terrain does not. A straight forward solutions is to have a dirty flag forcing all players to re-download the whole terrain and restart versioning. (Other solutions probably exists, but I think this is the best one since it does does not have any edge cases)
• Each region/block should have a sort of level of detail (LOD).
• Server should have two caches, one for reading voxels and one for writing.
• Reading buffer - When a read request, check buffer first, if not in buffer read requests terrain and its nearby terrain data and store in buffer (memory).
• Writing buffer - When writes requests are done, don't actually write at once but wait for some time possible save up some writes and do them all at once.
• Server should propagate changes in the terrain to all clients (at the moment, clients have to restart to get updated terrain).
• Server/Client should compress chunks before storing on disk (bzip2/gzip).
• Writing requests to the C module should be replicated to other areas that "run" the same area, making it scale horizontally. This allows servers to come out of "sync" with their terrain, a sane voting protocol (probably using the versioning if possible) would have to be implemented to allow server gone out of the data to sync up with the other servers.
• C module improvement could be done by some threading (I will try to avoid this as long as possible).

Not needed:
Clients does not need to request writes to the server side terrain since all logic run on the server. (For less "safer" solutions, this is just a matter of adding a new message type on the protocol).

Problems:
Server disk space not much of an issue, our plan is to have 10x10 areas of 8x8x8x -> 16x16x16 -> 32x32x32 voxels (256 GB of data per area in worst case). Which means 25 terra byte for the whole world.
Client side, this would of course be a problem, but worst case is highly unlikely to be achieved, most areas will likely have many homogenous parts, for example half the area would be air, other half dirt/rock and most of the mixed chunks would be the surface or mines.
One could have a special storage cache rules for the client, for example not allowing storing more than X GB of data on disk, freeing up space as new data is written (size could be set by the players themselves but should have a minimun value for sanity). This also requires clients to request syncing a full area since the logic on the server would have this client information.

I am actually implementing a voxel based terrain in our MMO (Abydos Online).

I wrote a simple C module, 1000 lines of code (500 for functionality and 500 unit/integration tests) that reads/write voxels to disk.

Here is how it works:
The terrain is called a volume.
A volume is divided into regions (default 8x8x8 regions).
One region is divided into blocks (default 16x616x16 blocks).
One block is 32x32x32 voxels (1 voxel = 1 cubic meter).
Each voxel is 4 bytes, 1 byte density, 2 byte material and 1 byte reserved.

When terrain is initiated, the C module creates a directory with one file per region. If the region is homogenous, only that file exists which contains the voxel for the whole region (for example all air, dirt, water etc). The filename would be "region_x_x_x.dat" where the x is replaced with the x, y and z position.

If a region is not homogenous, a directory "region_x_x_x" is created which contains the blocks, by default 32x32x32 blocks result in 4096 files, each of this file is named "block_x_y_z.dat". If this file will either be 1 voxel (4 bytes) in size of a homogenous block or 32x32x32 voxels (128 kb) for a mixed block.

A block with all the 128 kb of data is referred to as a chunk.

Currently working:
When a player logs in, the whole terrain is read and sent back to the player, regions and blocks are sent one as one TCP packet back to the client, chunks divided into parts of 8kb each and reassembled in client.
The client stores the data in the same structure as the server but also parses all regions/blocks/chunks and build static terrain geometry of this.

Todo:

• Terrain versioning, each region/block/chunk should have a version number (2bytes, 4 bytes, 8 bytes?) that is increased per change. At one point the max version will likely be reached and you will have edge cases where players that should download updated terrain does not. A straight forward solutions is to have a dirty flag forcing all players to re-download the whole terrain and restart versioning. (Other solutions probably exists, but I think this is the best one since it does does not have any edge cases)
• Each region/block should have a sort of level of detail (LOD).
• Server should have two caches, one for reading voxels and one for writing.
• Reading buffer - When a read request, check buffer first, if not in buffer read requests terrain and its nearby terrain data and store in buffer (memory).
• Writing buffer - When writes requests are done, don't actually write at once but wait for some time possible save up some writes and do them all at once.
• Server should propagate changes in the terrain to all clients (at the moment, clients have to restart to get updated terrain).
• Server/Client should compress chunks before storing on disk (bzip2/gzip).
• Writing requests to the C module should be replicated to other areas that "run" the same area, making it scale horizontally. This allows servers to come out of "sync" with their terrain, a sane voting protocol (probably using the versioning if possible) would have to be implemented to allow server gone out of the data to sync up with the other servers.
• C module improvement could be done by some threading (I will try to avoid this as long as possible).

Not needed:
Clients does not need to request writes to the server side terrain since all logic run on the server. (For less "safer" solutions, this is just a matter of adding a new message type on the protocol).

Problems:
Server disk space not much of an issue, our plan is to have 10x10 areas of 8x8x8x -> 16x16x16 -> 32x32x32 voxels (256 GB of data per area in worst case). Which means 25 terra byte for the whole world.
Client side, this would of course be a problem, but worst case is highly unlikely to be achieved, most areas will likely have many homogenous parts, for example half the area would be air, other half dirt/rock and most of the mixed chunks would be the surface or mines.
One could have a special storage cache rules for the client, for example not allowing storing more than X GB of data on disk, freeing up space as new data is written (size could be set by the players themselves but should have a minimun value for sanity). This also requires clients to request syncing a full area since the logic on the server would have this client information.

I am actually implementing a voxel based terrain in our MMO (Abydos Online).

I wrote a simple C module, 1000 lines of code (500 for functionality and 500 unit/integration tests) that reads/write voxels to disk.

Here is how it works:
The terrain is called a volume.
A volume is divided into regions (default 8x8x8 regions).
One region is divided into blocks (default 16x616x16 blocks).
One block is 32x32x32 voxels (1 voxel = 1 cubic meter).
Each voxel is 4 bytes, 1 byte density, 2 byte material and 1 byte reserved.

When terrain is initiated, the C module creates a directory with one file per region. If the region is homogenous, only that file exists which contains the voxel for the whole region (for example all air, dirt, water etc). The filename would be "region_x_x_x.dat" where the x is replaced with the x, y and z position.

If a region is not homogenous, a directory "region_x_x_x" is created which contains the blocks, by default 32x32x32 blocks result in 4096 files, each of this file is named "block_x_y_z.dat". If this file will either be 1 voxel (4 bytes) in size of a homogenous block or 32x32x32 voxels (128 kb) for a mixed block.

A block with all the 128 kb of data is referred to as a chunk.

Currently working:
When a player logs in, the whole terrain is read and sent back to the player, regions and blocks are sent one as one TCP packet back to the client, chunks divided into parts of 8kb each and reassembled in client.
The client stores the data in the same structure as the server but also parses all regions/blocks/chunks and build static terrain geometry of this.

Todo:
Terrain versioning, each region/block/chunk should have a version number (2bytes, 4 bytes, 8 bytes?) that is increased per change. At one point the max version will likely be reached and you will have edge cases where players that should download updated terrain does not. A straight forward solutions is to have a dirty flag forcing all players to re-download the whole terrain and restart versioning. (Other solutions probably exists, but I think this is the best one since it does does not have any edge cases)

Each region/block should have a sort of level of detail (LOD).
Server should have two caches, one for reading voxels and one for writing.
Reading buffer - When a read request, check buffer first, if not in buffer read requests terrain and its nearby terrain data and store in buffer (memory).
Writing buffer - When writes requests are done, don't actually write at once but wait for some time possible save up some writes and do them all at once.
Server should propagate changes in the terrain to all
Server/Client should compress chunks before storing on disk (bzip2/gzip).
Writing requests to the C module should be replicated to other areas that "run" the same area, making it scale horizontally. This allows servers to come out of "sync" with their terrain, a sane voting protocol (probably using the versioning if possible) would have to be implemented to allow server gone out of the data to sync up with the other servers.
C module improvement could be done by some threading (I will try to avoid this as long as possible).

Not needed:
Clients does not need to request writes to the server side terrain since all logic run on the server. (For less "safer" solutions, this is just a matter of adding a new message type on the protocol).

Problems:
Server disk space not much of an issue, our plan is to have 10x10 areas of 8x8x8x -> 16x16x16 -> 32x32x32 voxels (256 GB of data per area in worst case). Which means 25 terra byte for the whole world.
Client side, this would of course be a problem, but worst case is highly unlikely to be achieved, most areas will likely have many homogenous parts, for example half the area would be air, other half dirt/rock and most of the mixed chunks would be the surface or mines.
One could have a special storage cache rules for the client, for example not allowing storing more than X GB of data on disk, freeing up space as new data is written (size could be set by the players themselves but should have a minimun value for sanity). This also requires clients to request syncing a full area since the logic on the server would have this client information.