The water simulation we went with is not like anything I've read about. There were several methods I considered before going with what we have now.
First was particle water.
The pros: Good water simulation. Realistic waves, breaking etc. possible.
The cons: Hard to extract surface. Impossible to keep track of all particles on any significant planetary scale.
This was obviously not going to work for us.
Height-field based water.
The pros: Significantly less storage. Easy surface extraction. Decent water simulation.
The cons: Braking waves are harder (though not impossible). How do you do a height field based water on a spherical planet? The answer: not well. You can either split into 6 separate height fields, or try to create one with polar coordinates based on a even point distribution.
This is too bad because back before we went for an actual planet, on a flat 2d terrain, this was my top choice
What I went with:
Storing water in a 3d voxel density grid. Much like terrain.
Pros: Storage concerns were already figured out - storing can be done in same datablocks as we store terrain - thus its possible on a planetary scale.
Cons: It's not a very realistic simulation. It's hard to make huge waves.
There was also one other pro, which i didn't realize until later - updating water was made just somewhat easier by the fact that I stored water on a grid. Of course, the grid is NOT oriented with the surface, yet due to a range of densities [-127,127] - it was possible to achieve a perfectly smooth water surface anywhere on the planet despite the grid being all squirrly.
Here are some screenshots of the apparently misaligned grid and the non-the less smooth water surface:
And here is a video of the new water shader:
And a video with the older shader, but the only video of water spreading in a huge hole.
Update: video of the water on a small planet (200m radius)
For more info, and a demo of the project, you can visit at http://blog.milchopenchev.com.