I'm currently working on a cylinder shaped terrain produced by a height map. What happens in the program is simple, there is a texture for the colors of the terrain that has the alpha value of regions in with i want it to be invisible and another texture ARGB with the A being the gray scale for the heights and RGB is the normal for the light. The texture is such that the A value goes from 1 to 255 and I'm reserving the 0 for the regions with holes, meaning i don't want then to exist. So in theory no problem, I'm making those regions invisible based on the first texture but on practice what's happening is that the program is considering the 0 as the minimum height and, even with the texture on top, is creating some lines towards this regions of 0, like trying to make its triangle but not getting there because i cut the next vertex by making it invisible.

The image below is with a clip() function making the region invisible



This next image is without the clip function:



Notice the lines going to the center of the cylinder This is how it gets when i stop making those vertex invisible
So, just to say, i used the function Clip() on the pixel shader to make it invisible.

My questions are: Is it possible, the same way i used clip() on the pixel shader i do something like that on the vertex shader and get rid of the unwanted vertex? Basically, is possible to just say to ignore value 0? Any ideas to fix this? Another thing is that we can see that the program is interpolating the values from one vertex to the next, that is why i cuts on halfway through to the invisible vertex.

On a more technical point of view the domain shader is the one picking the heights from the texture and plotting it.

I managed to fix this somehow by modifying the texture, through the program, by changing the 0 values of the heights to a more proper one, but i'd like to know if there's a possibility through the HLSL.

I'm working with Directx11 API with C++ and the program uses Tessellation.
Thank you for your time and will be very glad with any input on this!

I dont think that is possible. The heightmap texture will be sampled in the Vertex Shader based on the geometry and interpolated for the triangle inners. To have a hole you'd have to have a hole in your geometry, such that the Vertex Shader would not get passed the triangles that describe the hole. I've never thought about that possibility - so I could be way out of line, but I've got a height map based terrain and I also would struggle to punch a hole in the geometry without doing it explicitly.

Alternatively you could just paint the Zero height region transparent - after all it only needs to look like a hole doesn't it ?

Well guys, first thank you very much for your input.

Based on the overall answer it looks like this "move" on the VS is a very tricky one and not recommended if it is possible.

And yes what i was doing is using a diffuse texture to get those holes transparent and what i was trying to fix was this weird lines trying to reach the regions of zero because from one vertex to the zero one there is a direct line connecting then and when i clip the zero vertex it keeps half the line.

As sad on my first post i managed to fix this quite well by changing the zeros of the texture to a better suited value. In a way that the zero region would now have a value equal to its non-zero neighbor and the lines wouldn't go weirdly to the center so when i passed the diffuse texture the lines would, in theory, be pointing along the the cylinder terrain.
What did happen is that i can hardly see the lines now, so great.

Thanks very much, it was very informative (btw i loved the L4D2 thing)