Jump to content

  • Log In with Google      Sign In   
  • Create Account

We're offering banner ads on our site from just $5!

1. Details HERE. 2. GDNet+ Subscriptions HERE. 3. Ad upload HERE.


Stencil test stop working when geometry shader is enabled


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
8 replies to this topic

#1 leonard2012   Members   -  Reputation: 167

Like
0Likes
Like

Posted 07 January 2013 - 10:02 AM

I use stencil test to restrict rendering of graphics objects to be inside a region. The works when geometry shader is set to NULL. However when I set the geometry shader, the stencil test seems to stop working for objects passed through geometry shader. and it only works for objects that does not pass throught the GS (e.g. texture). Could someone kindly tell me what's wrong.


Sponsor:

#2 Dan Mayor   Crossbones+   -  Reputation: 1714

Like
0Likes
Like

Posted 07 January 2013 - 03:14 PM

Debugging is your friend.  If the compiler isn't throwing any warnings or errors then your next step would be to "step" through your code.  MSVC has a GREAT debugger built in where you can add in break points at various places in your code (particularly all around where you believe the code to be failing).  When the program gets to that point in the code it will stop and highlight the line where you placed the breakpoint.  Now you can just hover over variables to see what they contain.  Look at what your values are (make sure you have a nice clean break before you encounter the problem so you know what you SHOULD expect).  Compare these values throughout the suspect code and it should give you an idea what happened.  Unfortunately I don't think we can give you an answer without more direct information.  "My programs broke, whats wrong?"  Could be many things, you could be invoking the shader incorrectly, you could be losing your object references in the process, you might be nulling something out, the shader itself might be working as designed but is breaking your geometry...  I could go on, but we can start with some basic debugging to see what the problem actually is.


Digivance Game Studios Founder:

Dan Mayor - Dan@Digivance.com
 www.Digivance.com


#3 Jason Z   Crossbones+   -  Reputation: 5282

Like
0Likes
Like

Posted 07 January 2013 - 08:20 PM

Are you able to take a frame capture with PIX (or the Graphics Debugger in VS2012)?  If so, that might help you determine what is actually happening in that draw call.  The geometry shader shouldn't have a bearing on the stencil tests, but it is possible that you are using multiple render target views which could have an impact.

 

Can you describe the rendering scenario (i.e. the pipeline connections to resources) when it doesn't work as expected?  Also, when you say that it stops working, do you mean that the stencil test is always failed, always passed, or something else?



#4 leonard2012   Members   -  Reputation: 167

Like
0Likes
Like

Posted 08 January 2013 - 01:07 AM

Debugging is your friend.  If the compiler isn't throwing any warnings or errors then your next step would be to "step" through your code.  MSVC has a GREAT debugger built in where you can add in break points at various places in your code (particularly all around where you believe the code to be failing).  When the program gets to that point in the code it will stop and highlight the line where you placed the breakpoint.  Now you can just hover over variables to see what they contain.  Look at what your values are (make sure you have a nice clean break before you encounter the problem so you know what you SHOULD expect).  Compare these values throughout the suspect code and it should give you an idea what happened.  Unfortunately I don't think we can give you an answer without more direct information.  "My programs broke, whats wrong?"  Could be many things, you could be invoking the shader incorrectly, you could be losing your object references in the process, you might be nulling something out, the shader itself might be working as designed but is breaking your geometry...  I could go on, but we can start with some basic debugging to see what the problem actually is.

Thank you for your quick reply. I've put many DX::ThrowIfFailed macro on most the D3D11 function calls. It seems that my code run fine as no exceptions are thrown. The only issue is that stencil test does not work for graphics objects that use geometry shader. For graphics objects that does not use geometry shader, everything is OK. And I don't know how to debug stencil test as no C++/HLSL code is directly related to stencil test.



#5 leonard2012   Members   -  Reputation: 167

Like
0Likes
Like

Posted 08 January 2013 - 01:32 AM

Are you able to take a frame capture with PIX (or the Graphics Debugger in VS2012)?  If so, that might help you determine what is actually happening in that draw call.  The geometry shader shouldn't have a bearing on the stencil tests, but it is possible that you are using multiple render target views which could have an impact.

 

Can you describe the rendering scenario (i.e. the pipeline connections to resources) when it doesn't work as expected?  Also, when you say that it stops working, do you mean that the stencil test is always failed, always passed, or something else?

 

Thanks for your suggestions.

For all graphics objects, vertex shader and pixel shader are used. For line object, geometry shader is added to implement line thickness. For objects (polygons, textures) that do not use geometry shader, stencil test works as expected. For line objects, if geometry shader is used, stencil test always passed. If geometry shader is not used, stencil test works as expected. 



#6 Jason Z   Crossbones+   -  Reputation: 5282

Like
0Likes
Like

Posted 08 January 2013 - 07:21 PM

That helps a little bit - now can you describe how you are using the stencil test?  For example, what type of stencil operations are you using?  From the description that you provided, it sounds like you may have your front face and back face operations set differently.  Especially if you are generating the geometry from the geometry shader, there is a possibility that you are producing back faces and don't realize it...  That could in turn then behave differently for the stencil test depending on what operations you have specified.



#7 leonard2012   Members   -  Reputation: 167

Like
0Likes
Like

Posted 09 January 2013 - 05:04 AM

Here is the code I create depth-stencil states. I maintains three states. One (m_depthstencilState) for normal conditions where stencil test is disabled. One (m_createMaskDSState) for filling stencil buffer with 1s inside the mask. And the third (m_applyMaskDSState) is for applying stencil test.

   // Create depth/stencil state
   D3D11_DEPTH_STENCIL_DESC dsDesc;
   ZeroMemory(&dsDesc, sizeof(D3D11_DEPTH_STENCIL_DESC));

   // Depth test parameters
   dsDesc.DepthEnable = FALSE;
   dsDesc.DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ALL;
   dsDesc.DepthFunc = D3D11_COMPARISON_LESS;

   // Stencil test parameters
   dsDesc.StencilEnable = FALSE;
   dsDesc.StencilReadMask = 0xFF;
   dsDesc.StencilWriteMask = 0xFF;

   // Stencil operations if pixel is front-facing
   dsDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_REPLACE;
   dsDesc.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS;

   // Stencil operations if pixel is back-facing
   dsDesc.BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.BackFace.StencilPassOp = D3D11_STENCIL_OP_REPLACE;
   dsDesc.BackFace.StencilFunc = D3D11_COMPARISON_ALWAYS;

   // Create depth stencil state
   DX::ThrowIfFailed(m_d3dDevice->CreateDepthStencilState(&dsDesc, &m_depthstencilState));

    dsDesc.DepthEnable = TRUE; // to experiment
   dsDesc.StencilEnable = TRUE;
   DX::ThrowIfFailed(m_d3dDevice->CreateDepthStencilState(&dsDesc, &m_createMaskDSState));

	dsDesc.DepthWriteMask   = D3D11_DEPTH_WRITE_MASK_ZERO;
	dsDesc.DepthFunc        = D3D11_COMPARISON_ALWAYS; 
   dsDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.FrontFace.StencilFunc = D3D11_COMPARISON_EQUAL;
   DX::ThrowIfFailed(m_d3dDevice->CreateDepthStencilState(&dsDesc, &m_applyMaskDSState));


#8 Nik02   Crossbones+   -  Reputation: 2915

Like
0Likes
Like

Posted 09 January 2013 - 06:31 AM

Following Jason's line of questions, show us your rasterizer state, and especially the cull property :)


Niko Suni


#9 leonard2012   Members   -  Reputation: 167

Like
0Likes
Like

Posted 09 January 2013 - 06:17 PM

Thank you very much to all of you. The issue results from the dis-match properties of rasterizer state and stencil state. I add two lines before creation of m_applyMaskDSState and fix the issue

   dsDesc.BackFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
   dsDesc.BackFace.StencilFunc = D3D11_COMPARISON_EQUAL;





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS