Sign in to follow this  
lomateron

stencil not working when rendering intances

Recommended Posts

lomateron    491

Has anyone had this problem?

 

I am rendering many point instances to random locations of a 2Dtexture(different points can land in the same texel), if i render them using drawInstance() the stencil buffer doesnt updates(a texel's stencil value is increased when the point lands on it)

if i draw them using a draw() inside a loop the stencil updates its value.

Edited by lomateron

Share this post


Link to post
Share on other sites
lomateron    491

I just found the root of my problem(not the solution) ...Weirdest of all i have ever had: 

for(int rrr=0;rrr!=1;rrr++)
{
	g_pd3dDevice->ClearDepthStencilView( g_pDepthStencilViewSpace3D, D3D10_CLEAR_STENCIL, 0.0f, 0 );
	g_pd3dDevice->OMSetDepthStencilState(g_pDepthStencilStateSpace3D,1);
	     if(rrr==0){g_pd3dDevice->OMSetBlendState(g_pBlSRed	        , 0, 0xffffffff);}
	else if(rrr==1){g_pd3dDevice->OMSetBlendState(g_pBlSGreen	, 0, 0xffffffff);}
	else if(rrr==2){g_pd3dDevice->OMSetBlendState(g_pBlSBlue	, 0, 0xffffffff);}
	else if(rrr==3){g_pd3dDevice->OMSetBlendState(g_pBlSAlpha	, 0, 0xffffffff);}

	g_pT->SetFloat((float)rrr);
	g_pTechnique3DSpace->GetPassByIndex( 0 )->Apply( 0 );
	g_pd3dDevice->DrawInstanced( 1, 1, 0, 0);
}

The principal problem is that its passing the stencil test in the 2nd, 3rd, 4th and on an on loop, and appart from that in those same loops there are other weird things happening (i don't know why) that may be related to why it passes the stencil test.

 

I am rendering to 1 texel of a texture using a point vertex

The default value of all texels in texture is float4(-1,-1,-1,-1)

The stencil test passes if EQUAL, OMSetDepthStencilState() sets to 1 and ClearDepthStencilView() sets the other to 0, it will have to fail always

the depth test is disabled, if depth test fails nothing happens

if the stencil test passes or fails the stencil value is incremented

The blending just mask a color(the color in the name)

The value rendered to the texel is float4(rrr,rrr,rrr,rrr)

 

If the loop does 1 loop: for(int rrr=0;rrr!=1;rrr++)

-nothing is write to the texel, it failed the stencil test.

 

if the loop does 2 loops: for(int rrr=0;rrr!=2;rrr++)

-In the first loop nothing is rendered, it failed the stencil test.

-In the second loop it passes the stencil test(I dont know why) so it renders to the texel float4(0,0,0,0)-->it should be float4(1,1,1,1) but its not (i don't know why)

and it is masked by the unupdated red mask(i dont know why)-->it should be the green mask, the final texel value is float4(0,-1,-1,-1)-->g_pT is 0 not 1(i don't know why)

 

if the loop does 3 loops: for(int rrr=0;rrr!=3;rrr++)

-In the first loop nothing is rendered, it failed the stencil test.

-In the second loop it passes the stencil test(I don't know why), it renders to the texel float4(0,0,0,0)-->it should be float4(1,1,1,1) but its not (i don't know why)

and  it is masked by the red mask(i don't know why)-->it should be the green mask, the final texel value is float4(0,-1,-1,-1)-->g_pT is 0 not 1(i don't know why)

-In the third loop it passes the stencil test(I don't know why) so it renders to the texel float4(1,1,1,1) -->it should be float4(2,2,2,2) but its not (i don't know why)

and  it is masked by the green mask(i don't know why)-->it should be the blue mask, the final texel value is float4(0,1,-1,-1)

-->g_pT is 1 not 2(i don't know why)

 

In the end if i replace drawInstance() by a draw() inside a loop it all runs correctly but i cant because in the real code i run aroun 5000 instances

Edited by lomateron

Share this post


Link to post
Share on other sites
Seabolt    781

Should you by clearing the DepthStencilView each iteration of the loop? Everytime you clear your entire stencil buffer, and you'll never have anything to test it with, (essentially you're only testing against 0). Not to mention it's terribly slow.

Share this post


Link to post
Share on other sites
lomateron    491

I know that its going to happen, i made it that way, its a test code that shows how i discovered the root of my problem, and the problem is that it passes in the second loop the stencil test when it shoul never ever pass it, but its passing the test and i want to know why it is passing the test in the 2nd, 3rd, 4th, and on and on, that's the weird problem i am refering to.

Edited by lomateron

Share this post


Link to post
Share on other sites
Seabolt    781

I can't honestly say I completely understand what's happening here. It seems like if it's passing the stencil test on the 2nd, 3rd, and 4th iterations, then you're not writing what you expect into the stencil buffer. Break out PIX, (or equivalent), and make sure all the states are where you think that they should be, and that the stencil buffer is what you think it should be.

Share this post


Link to post
Share on other sites
Cthuga    132

I'm trying to understand Your situation.

 

I suppose that Your DepthStencilBuffer is enabled, his test is LESS THAN (standard one), and it MASK_ALL - so You make standard depth stencil test. Am I right ?

 

Could You explain am I right:

 

The stencil test passes if EQUAL, OMSetDepthStencilState() sets to 1 and ClearDepthStencilView() sets the other to 0,

 

So You want to set Stencil buffer to 1 there where You draw Your Instance ?

 

But then You wrote:

 

if the stencil test passes or fails the stencil value is incremented

 

So I see here two different things: Do You set it to 1, or increment in every iteration ?

 

______________________________________________________________________________________________________________

 

Geek Message Blog

The steps I take to make a dream come true:

 

http://geekmessage.wordpress.com

Edited by Cthuga

Share this post


Link to post
Share on other sites
lomateron    491

I think I just found a directx bug, can someone try this to see if its really a bug, it's simple:

 

create a texture with 1 texel....create it's render target..create its depthstencil with this DXGI_FORMAT_D24_UNORM_S8_UINT

create depthstencil state with this description:

D3D10_DEPTH_STENCIL_DESC dSDd;
dSDd.DepthEnable = FALSE;
dSDd.DepthWriteMask = D3D10_DEPTH_WRITE_MASK_ZERO;
dSDd.DepthFunc = D3D10_COMPARISON_ALWAYS;

dSDd.StencilEnable = TRUE;
dSDd.StencilReadMask = D3D10_DEFAULT_STENCIL_READ_MASK;
dSDd.StencilWriteMask = D3D10_DEFAULT_STENCIL_WRITE_MASK;

dSDd.FrontFace.StencilDepthFailOp	= D3D10_STENCIL_OP_KEEP;
dSDd.FrontFace.StencilFailOp		= D3D10_STENCIL_OP_INCR;
dSDd.FrontFace.StencilPassOp		= D3D10_STENCIL_OP_INCR;
dSDd.FrontFace.StencilFunc		= D3D10_COMPARISON_EQUAL;

dSDd.BackFace.StencilDepthFailOp	= D3D10_STENCIL_OP_KEEP;
dSDd.BackFace.StencilFailOp		= D3D10_STENCIL_OP_KEEP;
dSDd.BackFace.StencilPassOp		= D3D10_STENCIL_OP_KEEP;
dSDd.BackFace.StencilFunc		= D3D10_COMPARISON_NEVER;

 

create 2 techniques with this shaders, first technique uses VS1 and second technique uses VS2

 

 

struct VS_A
{
    float4 Pos : POSITION;
    uint Id : SV_InstanceID;
};

struct PS_A
{
    float4 Pos : SV_POSITION;
    float4 Texl : TTTTT;
};

PS_A VS1( VS_A i  )//stencil will not work as it should
{
	PS_A output;

	output.Pos=float4(0.0f,0.0f,0.0f,1.0f);
	output.Texl=float4(1.0f,1.0f,1.0f,1.0f);

	return output;
}

PS_A VS2( float4 Pos : POSITION  )//stencil will work correctly
{
	PS_A output;

	output.Pos=float4(0.0f,0.0f,0.0f,1.0f);
	output.Texl=float4(1.0f,1.0f,1.0f,1.0f);

	return output;
}

float4 PS( PS_A a) : SV_Target
{
	return a.Texl;
}

 

 

now to render

set the render target with its depthstencil, set its viewport and layout and finally render it like this:

 

g_pd3dDevice->OMSetDepthStencilState(g_pDepthStencilState,1);

g_pd3dDevice->ClearDepthStencilView( g_pDepthStencilViewSpace3D, D3D10_CLEAR_STENCIL, 0.0f, 0 );
g_pTechnique1->GetPassByIndex( 0 )->Apply( 0 );
g_pd3dDevice->DrawInstanced( 1, 1, 0, 0);

g_pd3dDevice->ClearDepthStencilView( g_pDepthStencilViewSpace3D, D3D10_CLEAR_STENCIL, 0.0f, 0 );
g_pTechnique1->GetPassByIndex( 0 )->Apply( 0 );
g_pd3dDevice->DrawInstanced( 1, 1, 0, 0);

 

 

 

the first draw will fail the stencil test and will not draw nothing,

the second draw will pass the stencil test and will draw ----------->This is the problem, i don't know why it passes the test

 

Now change the technique so it uses the VS2 and render using: g_pd3dDevice->Draw( 1, 0);

The two draws will fail the stencil test, thats how it should work

 

The use of SV_Instance in VS1 i think is the cause of the problem, i dont know more

Edited by lomateron

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this