I have Z-pass working for my D3D11 application but no matter what I try I can't get z-fail to work. Here is my depth stencil state for building the stencil buffer with z-pass:

D3D11_DEPTH_STENCIL_DESC depthStencilDesc = {0};
	depthStencilDesc.DepthEnable = true;
	//Disable depth writes when rendering the volumes
	depthStencilDesc.DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ZERO;
	depthStencilDesc.DepthFunc = D3D11_COMPARISON_LESS;
	depthStencilDesc.StencilEnable = true;
	depthStencilDesc.StencilReadMask = D3D11_DEFAULT_STENCIL_READ_MASK;
	depthStencilDesc.StencilWriteMask = D3D11_DEFAULT_STENCIL_WRITE_MASK;

	//Stencil operation for front-facing polygons should increment
	depthStencilDesc.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
	depthStencilDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
	depthStencilDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_INCR;
	depthStencilDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
	

	//Stencil operation for back-facing polygons should decrement
	depthStencilDesc.BackFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
	depthStencilDesc.BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
	depthStencilDesc.BackFace.StencilPassOp = D3D11_STENCIL_OP_DECR;
	depthStencilDesc.BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
	

	hr = g_pd3dDevice->CreateDepthStencilState(&depthStencilDesc, &buildShadowVolumeDepthStencilState);

How would I modify this to do z-fail?

That makes sense but it doesn't work. It makes the floor black except where geometry is behind the floor. Don't I need to change DepthFunc to GREATER? I want only fragments behind the depth map to be counted. Even when I try that it doesn't work.

Yeah I'm doing all of those things. Like I said, Z-pass works. That implies that I have culling disabled. I use a stencil ref of 127 and clear to that as well. It may be worth using the saturate option for the stencil operation but it shouldn't be a problem for my simple test case.

I've been trying to get this for weeks and I just can't figure it out. There has to be something going on behind the scenes that I'm missing.

I've attached a picture of Z-pass working and what happens when I try z-fail using StencilDepthFailOp = INCR/DECR and DepthFunc = LESS. Note that StencilDepthFailOp is KEEP for z-pass and StencilPassOp is INCR/DECR instead.

The third image is when I set DepthFunc to GREATER_EQUAL and StencilDepthFailOp to INCR/DECR. This should be another method of z-pass because I have inverted the depth function. Now values that are less than the depth value will fail the depth test and cause the stencil value to change. But I can't explain the squares of grey pixels around the chair. Is D3D doing the depth test at a lower resolution, then using that decision for a whole square of fragments? I can't think of any other reason why those little grey squares would appear.

One place where I am confused by the API: what happens if the depth AND the stencil tests fail? There is a case for the depth fail operation and a case for the stencil fail operation. Which one occurs?

I appreciate the replies mhagain.

Solved it! My geometry shader was producing front caps and back caps with inverted normals. This caused the stencil count to be off because front faces increment and back faces decrement.

For future reference, the way mhagain set up the DepthStencil is the correct way. The front-facing shadow polygons could increment and the back-facing polygons could decrement, but it doesn't matter. The depth test should be LESS, so that failures are equal to or greater than the depth buffer.

The accidental inverted normal is like the missed decimal place for graphics programmers: