The problem is, that every terrain system will implement this on their own - and it will differ, an example from me: as height map is technically a plane - I decided to implement binary triangle trees.

To explain the image - at the center of tessellated mesh there is an observer, you can see terrain cells being tessellated at different level. As the implementation is done using binary triangle trees - adjacency is guaranteed (assuming you choose correct index buffer).

In terms of LOD I can't get less tessellation than 2 triangles in cell.

Supported height map is of "unlimited" size, as my height map is a virtual texture (the page requests are not based on what you see like in Rage or Quake Wars (where they used virtual textures for albedo, normals, etc.), but based on selected tessellation level on the terrain). I had to solve problem with authoring terrains by building custom editor.

When editing the height map of the terrain, you always edit the page on most detailed pyramid level (and higher pyramid levels are then computed).

Thsi is WIP, but: during editing of height map I allow for specifying a 'hole' (basically negative values as of now), which is then written into the virtual texture (and all impacted pages are updated). As the update algorithm is technically a mip-map generation, I could always copy negative sign from child pages into parent one for given pixel - and therefore keep a hole even in the distance.

As for the games (like Skyrim) I'm quite sure they do render holes only on the most detailed terrain cell where you are currently at. Further they mask it with distant 3D model, so you never see the actual hole in the terrain.

Note. By pyramid, I mean pyramid of full virtual texture - when editing it - you need to load pages, edit/update them, and store in virtual texture file - as there is no way you could hold everything in memory at once.

Yes, that is way how I deal with holes still being in terrain in distance currently. As you pointed out, it can become bigger (which could be a problem, but so far I didn't notice it - of course this always boils down to: what kind of art are you using and whether you also place distant LOD geometry of your objects around the hole).

I think most of the games won't deal with them in distance (at least I'm quite sure none of Bethesda games show you holes only in loaded terrain cells around you).

Also, how do you render your terrain?