I am doing some terrain rendering and I've run in the some troubles. At this point in time I am just tessellating vertex patches then displacing them with a height map. My current problem is that the rendering looks pretty funky. I've been debugging this for a while and it looks like it is a problem with the depth buffer. Past that, I don't really have any ideas on what is happening.
Here's what visual studio says is going on with the depth buffer (different scene). Wasnt really sure the best way to show this using the debugger so I grabbed an image of the red channel and the green channel
Note: The top left of this image is actually a similar depth to the bottom area of the image
Here's backbuffer:
As I understanding it, closer pixels should be colored darker than further pixels. As you can see when comparing the second and third images, it seems like the opposite is happening. Am i misunderstanding something or is something weird happening.
Any thoughts or pointing in a direction would be very appreciated since I'm out of ideas.
I'll also post how I am setting up my depth buffer for possible questions:
D3D11_DEPTH_STENCIL_DESC dsdesc;
ZeroMemory(&dsdesc, sizeof(dsdesc));
dsdesc.DepthEnable = true;
dsdesc.DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ALL;
dsdesc.DepthFunc = D3D11_COMPARISON_LESS;
dsdesc.StencilEnable = true;
dsdesc.StencilReadMask = 0xFF;
dsdesc.StencilWriteMask = 0xFF;
dsdesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
dsdesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_INCR;
dsdesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
dsdesc.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
dsdesc.BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
dsdesc.BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_DECR;
dsdesc.BackFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
dsdesc.BackFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
HR(_device->CreateDepthStencilState(&dsdesc, &_depthStencilState));
D3D11_RASTERIZER_DESC rd;
rd.AntialiasedLineEnable = false;
rd.CullMode = D3D11_CULL_BACK;
rd.DepthClipEnable = true;
rd.DepthBias = 0;
rd.DepthBiasClamp = 0.0f;
rd.FillMode = D3D11_FILL_SOLID;
rd.FrontCounterClockwise = false;
rd.MultisampleEnable = false;
rd.ScissorEnable = false;
rd.SlopeScaledDepthBias = 0.0f;
