1 minute ago, JohnnyCode said:I adviced to render the cropping quad with transparent/opaque pixels as only stencil mask write , and then into that buffer render full screen cropped texture that will result in only the croped area being processed/rendered.
In my experience, it will be ding-fast.
My issue though is i'm rendering the area I want stenciled to a quad, but I don't want the whole quad as part of the stencil so the texture has to be taken into account.
1 minute ago, pindrought said:My issue though is i'm rendering the area I want stenciled to a quad, but I don't want the whole quad as part of the stencil so the texture has to be taken into account.
Transparent pixels will not pass to stencil/depth write.
Stencil buffer is evolving memory through entire frame rendering, so if set up correctly (cleared with a value), and with correct stencil test and correct writes, you should encounter no problem about that?
Imagine that at particular point (rendering crop before cropped image) you set a write mask, and on succesive call you set to pass only on that mask, beign sure mask value is not anywhere else. Should suit your case if I understand.
35 minutes ago, JohnnyCode said:Transparent pixels will not pass to stencil/depth write.
Stencil buffer is evolving memory through entire frame rendering, so if set up correctly (cleared with a value), and with correct stencil test and correct writes, you should encounter no problem about that?
Imagine that at particular point (rendering crop before cropped image) you set a write mask, and on succesive call you set to pass only on that mask, beign sure mask value is not anywhere else. Should suit your case if I understand.
Transparent pixels do pass to the stencil/depth write in DX11.
I ended up going with the discard method and it worked out.
Thanks all for the input.
Edit to Add: I appreciate the input JohnnyCode but I still don't understand how your method would work. The write mask is just which bits can be accessed for the stencil buffer operation correct? The actual transparency of my pixel should not have any effect on this when drawing the initial mask, or so I would think.
I don't know directx or anything, but if you are looking to do masking, why not just do bitwise masking on each pixel. This way you could create a layer that is your mask, this layer when masked with your true layer would transform the unwanted pixel to the desired color, and let the others pixel stay as they were.
Pixel color rgba are normally represented as 32 bits so 4 8bits 255,255,255,255.
But again, it may be too weird to do in directx.
1 hour ago, LLachance said:I don't know directx or anything, but if you are looking to do masking, why not just do bitwise masking on each pixel. This way you could create a layer that is your mask, this layer when masked with your true layer would transform the unwanted pixel to the desired color, and let the others pixel stay as they were.
Pixel color rgba are normally represented as 32 bits so 4 8bits 255,255,255,255.
But again, it may be too weird to do in directx.
As far as I know when you mask in the stencil buffer it is masking with the stencil value which is not representative of the actual pixel color or transparency, but is instead an 8 bit value (0-255) which you could use for many things.
I am currently using two depth stencils for what i'm achieving and two pixel shaders.
Each frame, I am clearing the depth stencil view and initializing the stencil value to 0.
deviceContext->ClearDepthStencilView(depthStencilView.Get(), D3D11_CLEAR_DEPTH | D3D11_CLEAR_STENCIL, 1.0f, 0);For when I draw the pink shaded area, I am using this pixel shader to discard the transparent pixels and use black for the background.
struct PS_INPUT
{
float4 inPosition : SV_POSITION;
float2 inTexCoord : TEXCOORD;
};
Texture2D objTexture : TEXTURE: register(t0);
SamplerState objSamplerState : SAMPLER: register(s0);
float4 main(PS_INPUT input) : SV_TARGET
{
float4 sampleColor = objTexture.Sample(objSamplerState, input.inTexCoord);
if (sampleColor.a < 0.25f) discard;
return float4(0.0f, 0.0f, 0.0f, 1.0f);
}
For the first depth stencil, when I draw the pink shaded area in the example, I am using this depth stencil description.
CD3D11_DEPTH_STENCIL_DESC depthstencildesc_maskpredraw(D3D11_DEFAULT);
depthstencildesc_maskpredraw.DepthEnable = FALSE;
depthstencildesc_maskpredraw.StencilEnable = TRUE;
//never do backface comparison as we only care about front facing triangles
depthstencildesc_maskpredraw.BackFace.StencilFunc = D3D11_COMPARISON_FUNC::D3D11_COMPARISON_NEVER;
//always do front face (no comparison when drawing the initial mask area)
depthstencildesc_maskpredraw.FrontFace.StencilFunc = D3D11_COMPARISON_FUNC::D3D11_COMPARISON_ALWAYS;
depthstencildesc_maskpredraw.FrontFace.StencilPassOp = D3D11_STENCIL_OP::D3D11_STENCIL_OP_INCR_SAT; //for the pixels in stencil buffer, increment and saturate stencil value
depthstencildesc_maskpredraw.StencilWriteMask = 0xFF;
depthstencildesc_maskpredraw.StencilReadMask = 0xFF;This way all of the pixels that are drawn should have a stencil value of 1 since they should've been incremented once.
For the second depth stencil where I draw the actual scene objects, I am using this depth stencil with a stencil ref value of 0 so that only pixels where the stencil ref < stencil value will be drawn which should be the pixels drawn to when I drew the shaded mask area.
//Create depth stencil state
CD3D11_DEPTH_STENCIL_DESC depthstencildesc_maskpostdraw(D3D11_DEFAULT);
depthstencildesc.DepthFunc = D3D11_COMPARISON_FUNC::D3D11_COMPARISON_LESS_EQUAL;
depthstencildesc_maskpostdraw.StencilEnable = TRUE;
//don't care about back facing triangles so never perform comparison
depthstencildesc_maskpostdraw.BackFace.StencilFunc = D3D11_COMPARISON_FUNC::D3D11_COMPARISON_NEVER;
//we only will pass stencil function if the stencil ref value is less than stencil buffer value
depthstencildesc_maskpostdraw.FrontFace.StencilFunc = D3D11_COMPARISON_FUNC::D3D11_COMPARISON_LESS;
depthstencildesc_maskpostdraw.FrontFace.StencilPassOp = D3D11_STENCIL_OP::D3D11_STENCIL_OP_KEEP;
depthstencildesc_maskpostdraw.FrontFace.StencilFailOp = D3D11_STENCIL_OP::D3D11_STENCIL_OP_KEEP;
depthstencildesc_maskpostdraw.StencilWriteMask = 0xFF;
depthstencildesc_maskpostdraw.StencilReadMask = 0xFF;
Edited to Add:
For reference, here is the mask/example that I am currently seeing in my project.
17 hours ago, pindrought said:Transparent pixels do pass to the stencil/depth write in DX11.
That is in the case of alpha blending not in the case of alpha masking.
10 hours ago, JohnnyCode said:That is in the case of alpha blending not in the case of alpha masking.
How would I do that using the stencil buffer?
12 hours ago, pindrought said:How would I do that using the stencil buffer?
No, you first render your croping quad of transparent zone with stencil write enabled and alpha masking enabled, this will set-up stencil buffer memory. Then you render your cropped image with stencil function to pass on the rendered stencil mask only.
9 minutes ago, JohnnyCode said:No, you first render your croping quad of transparent zone with stencil write enabled and alpha masking enabled, this will set-up stencil buffer memory. Then you render your cropped image with stencil function to pass on the rendered stencil mask only.
I'm still not understanding what you mean by alpha masking enabled and I can't find any examples regarding alpha masking in dx11. I've tried setting up a blend state that only takes alpha into account, but i'm still getting the same result as before where it seems like the transparency is being completely ignored by the stencil buffer.
46 minutes ago, pindrought said:I've tried setting up a blend state that only takes alpha into account, but i'm still getting the same result as before where it seems like the transparency is being completely ignored by the stencil buffer.
That is strange if you have proper value in your transparent texture. Try discarding pixels manualy when you render it upon its alpha comparison then? Just as a fast test to bypass around your alpha settings that you should publish.
