Like many others, I'm currently going through tool creation hell!
Part of that includes a heightmap editor which I'd gotten quite a way into - but it was CPU based and I was really unhappy with it. So I've deleted it, lock stock and barrel, and I'm going to rethink the whole thing and do it on the GPU.
Part of the problem was to do with vertex picking, and applying texture based brushes centred on the 'picked' vertex. Whilst quadtrees are ideal for this, as the map grows (and I want very large maps) edits can become quite a problem as the quad tree is constantly being updated before I can use it to get the vertex currently under the brush, and of course I have to guarantee that the data on the CPU & GPU are in sync. There where several other minor issues which also made the performance inconsistent.
My idea is to essentially write a traditional event driven Win32 program (GetMessage, vsync off, vs PeekMessage, vsync on) which triggers the GPU to update the heightmap on 'mouse down' events at fixed time intervals - maybe 20 times per second. For vertex picking, I intend to write to two colour buffers - a normal colour buffer, and a 32-bit RG floating point buffer using colours interpolated from 0.0f, 0.0f to heighmap width and height (e.g. 8192.0f). On mouse move, I can then use the mouse XY coordinates to copy the pixel value under the mouse into a 1x1 texture, which can then be read back by a shader (rounding the RG colour to the nearest integers will give me a precise XY vertex), which in turn can modify heighmap values based around these coordinates using a another texture 'brush'. This also allows for a really simple undo/redo mechanism totally on the GPU by pre-copying an area into a buffer before modifying the original data.
The benefits include only using one GPU side set of data, and better performance to boot.
I spent many, many weeks on the original version, so I thought it wise to post here before I embark on version 2!
My question is, can anyone think of any problems I may encounter with the above approach, or maybe suggest a totally different approach to a heightmap editor?