i made a terrain editor like that a while ago

i just cast a ray & checked for intersection with the triangles of the terrain

the closest triangle intersected was chosen and the point of intersection was used to find the closest vertex on the triangle

that vertex was the start point of whatever effect you were using, the hill effect for example was something like

for each vertex

compute distance from start vertex

divide by range to put in (0,1) interval

then plug that into cosine and scale

then use that value to increase or decrease the vertex height

i used a dynamic vertex buffer & used a uniform grid as my spatial partitioning structure to speed it all up

if i was gonna make another one i'd use vertex texture fetches to offset the vertices of a triangle grid right in the vertex shader & just modify the heightmap

Firstly, what is your target GPU? That, to a degree, will dictate how you approach this. A powerful desktop GPU with plenty of memory would suggests a GPU approach - a slow laptop GPU with limited memory may force a CPU approach.

Secondly, check out Scape. If you download it, you can use either a CPU GPU brushes to edit the heightmap. The GPU version is orders of magnitude faster (meaning far more fluid and reactive to user input).

Sorry for the late reply! Here's one way of editing heightmap data:

Create you heighmap texture, say 4097x4097 in VRAM. Call this the terrain texture.

Now decide on the maximum brush size you want to use, say 1025x1025 (you only need to use a small region of this for small brushes). Create *two* textures of this size. Call the first the brush texture, and the second the destination texture.

Ideally, use R32_FLOAT texture formats for all three textures, because you can store any unclamped floating point values (including negative).

Assuming you've got the world location of the mouse, render a quad into the destination texture the size of the brush, using the region from the heightmap and the region from the brush as texture inputs, add/subtract the values (and set min/max values for plateau's and canyons etc.). Now copy the contents of the destination back to the heightmap, and render.

In summary, if I set a brush size of 9 pixels, I'd create a 9x9 fp32 brush texture on the cpu and sub-copy it to the top left of the brush texture. I'd then draw a 9x9 quad to the destination texture using a 9x9 region from the heightmap centred on the mouse plus the 9x9 region from the brush. Do whatever math is appropriate, and then copy the 9x9 region from the destination texture back to the heightmap texture. Render the scene using the new heightmap data.