Debugging is your friend. If the compiler isn't throwing any warnings or errors then your next step would be to "step" through your code. MSVC has a GREAT debugger built in where you can add in break points at various places in your code (particularly all around where you believe the code to be failing). When the program gets to that point in the code it will stop and highlight the line where you placed the breakpoint. Now you can just hover over variables to see what they contain. Look at what your values are (make sure you have a nice clean break before you encounter the problem so you know what you SHOULD expect). Compare these values throughout the suspect code and it should give you an idea what happened. Unfortunately I don't think we can give you an answer without more direct information. "My programs broke, whats wrong?" Could be many things, you could be invoking the shader incorrectly, you could be losing your object references in the process, you might be nulling something out, the shader itself might be working as designed but is breaking your geometry... I could go on, but we can start with some basic debugging to see what the problem actually is.

Thank you for your quick reply. I've put many DX::ThrowIfFailed macro on most the D3D11 function calls. It seems that my code run fine as no exceptions are thrown. The only issue is that stencil test does not work for graphics objects that use geometry shader. For graphics objects that does not use geometry shader, everything is OK. And I don't know how to debug stencil test as no C++/HLSL code is directly related to stencil test.

Are you able to take a frame capture with PIX (or the Graphics Debugger in VS2012)? If so, that might help you determine what is actually happening in that draw call. The geometry shader shouldn't have a bearing on the stencil tests, but it is possible that you are using multiple render target views which could have an impact.

Can you describe the rendering scenario (i.e. the pipeline connections to resources) when it doesn't work as expected? Also, when you say that it stops working, do you mean that the stencil test is always failed, always passed, or something else?

Thanks for your suggestions.

For all graphics objects, vertex shader and pixel shader are used. For line object, geometry shader is added to implement line thickness. For objects (polygons, textures) that do not use geometry shader, stencil test works as expected. For line objects, if geometry shader is used, stencil test always passed. If geometry shader is not used, stencil test works as expected.

Here is the code I create depth-stencil states. I maintains three states. One (m_depthstencilState) for normal conditions where stencil test is disabled. One (m_createMaskDSState) for filling stencil buffer with 1s inside the mask. And the third (m_applyMaskDSState) is for applying stencil test.

   // Create depth/stencil state
   D3D11_DEPTH_STENCIL_DESC dsDesc;
   ZeroMemory(&dsDesc, sizeof(D3D11_DEPTH_STENCIL_DESC));

   // Depth test parameters
   dsDesc.DepthEnable = FALSE;
   dsDesc.DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ALL;
   dsDesc.DepthFunc = D3D11_COMPARISON_LESS;

   // Stencil test parameters
   dsDesc.StencilEnable = FALSE;
   dsDesc.StencilReadMask = 0xFF;
   dsDesc.StencilWriteMask = 0xFF;

   // Stencil operations if pixel is front-facing
   dsDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_REPLACE;
   dsDesc.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS;

   // Stencil operations if pixel is back-facing
   dsDesc.BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.BackFace.StencilPassOp = D3D11_STENCIL_OP_REPLACE;
   dsDesc.BackFace.StencilFunc = D3D11_COMPARISON_ALWAYS;

   // Create depth stencil state
   DX::ThrowIfFailed(m_d3dDevice->CreateDepthStencilState(&dsDesc, &m_depthstencilState));

    dsDesc.DepthEnable = TRUE; // to experiment
   dsDesc.StencilEnable = TRUE;
   DX::ThrowIfFailed(m_d3dDevice->CreateDepthStencilState(&dsDesc, &m_createMaskDSState));

	dsDesc.DepthWriteMask   = D3D11_DEPTH_WRITE_MASK_ZERO;
	dsDesc.DepthFunc        = D3D11_COMPARISON_ALWAYS; 
   dsDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.FrontFace.StencilFunc = D3D11_COMPARISON_EQUAL;
   DX::ThrowIfFailed(m_d3dDevice->CreateDepthStencilState(&dsDesc, &m_applyMaskDSState));

Thank you very much to all of you. The issue results from the dis-match properties of rasterizer state and stencil state. I add two lines before creation of m_applyMaskDSState and fix the issue

   dsDesc.BackFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.BackFace.StencilFunc = D3D11_COMPARISON_EQUAL;