XNA Stencil Buffer problem, with code

Graphics and GPU Programming Programming

Started by danromeo September 01, 2014 07:47 PM

3 comments, last by danromeo 9 years, 7 months ago

danromeo

238

Author

September 01, 2014 07:47 PM

hi.

I'm trying to create a simple stencil mask on a RenderTarget2D from a bunch of primitives, and then later draw pixels from that render target to another rendertarget in a shader based on the stencil test pass/fail. Code is below. The results that I'm getting seems to be that the stencil test either always passes every pixel or always fails every pixel, regardless of which settings I try for DepthStencilStates.

The idea is to create an overhead view of a forested world and then lay that view over the terrain when viewed from overhead rather than redrawing the forests on every frame, BUT my question is about Stencil Buffers..

I set up the following resources:


MyRenderTarget = new RenderTarget2D(graphicsDevice, mapSize, mapSize, true, SurfaceFormat.Color, DepthFormat.Depth24Stencil8, 0, RenderTargetUsage.DiscardContents);


NewRenderTarget = new RenderTarget2D(graphicsDevice, mapSize, mapSize, true, SurfaceFormat.Color, DepthFormat.Depth24Stencil8, 0, RenderTargetUsage.DiscardContents);

DepthStencilState writeStencil = new DepthStencilState()
                {
                    StencilEnable = true,
                    DepthBufferEnable = false,
                    ReferenceStencil = 1,
                    StencilFunction = CompareFunction.Always,
                    StencilPass = StencilOperation.Replace,
                };

            DepthStencilState stencilMask = new DepthStencilState()
            {
                StencilEnable = true,
                DepthBufferEnable = false,
                ReferenceStencil = 0,
                StencilFunction = CompareFunction.NotEqual,
                StencilPass = StencilOperation.Keep,
            };

During initialization to create my overhead render target with stencil I set the DepthStencilState to stencilMask and draw the forests to the rendertarget, which SHOULD give me a stencil buffer containing 0's where there are no trees and 1's where there are trees.


graphicsDevice.SetRenderTarget(MyRenderTarget);
graphicsDevice.Clear(ClearOptions.DepthBuffer | ClearOptions.Stencil | ClearOptions.Target, Microsoft.Xna.Framework.Color.Black, 1.0f, 0);


graphicsDevice.DepthStencilState = writeStencil;

            foreach (EffectPass pass in effect.CurrentTechnique.Passes)
            {
                pass.Apply();

                graphicsDevice.DrawUserIndexedPrimitives<Position4Texture>(PrimitiveType.TriangleList,
                   Vertices, 0, 4, Indices, 0, 2);

            }
graphicsDevice.DepthStencilState = DepthStencilState.Default;

And then at render time I render my terrain, and then in a second pass I set the DepthStencilState to stencilMask and render a quad over the terrain pulling pixels from MyRenderTarget based on stencil test pass/fail:


graphicsDevice.SetRenderTarget(NewRenderTarget);
graphicsDevice.Clear(ClearOptions.DepthBuffer | ClearOptions.Stencil | ClearOptions.Target, Microsoft.Xna.Framework.Color.Black, 1.0f, 0);

graphicsDevice.DepthStencilState = DepthStencilState.Default;
< DRAW TERRAIN TO NewRenderTarget >


graphicsDevice.DepthStencilState = stencilMask;
effect.Parameters["Texture"].SetValue(MyRenderTarget);

            foreach (EffectPass pass in effect.CurrentTechnique.Passes)
            {

                pass.Apply();

                graphicsDevice.DrawUserIndexedPrimitives<Position4Texture>(PrimitiveType.TriangleList,
                   Vertices, 0, 4, Indices, 0, 2);

            }
            graphicsDevice.DepthStencilState = DepthStencilState.Default;

And in the simple pixel shader I am returning:


return = tex2D(Texture,input.TexCoord);

I've tried various settings in the DepthStencilStates, and the end result is the stencil test either always passes all pixels, giving me overhead forests with black terrain, or always fails, giving me terrain with no forests. I've never used stencil buffers before but would like to make extensive use of them. Can somebody tell me what I'm doing wrong?

THANKS

phil_t

8,087

September 02, 2014 04:54 PM

I already answered this for you: http://www.gamedev.net/topic/660306-xna-4-rendertarget-stencil-buffer/#entry5176471

Cascade Quest - IceFall Games - Dev blog - twitter - youtube

danromeo

238

Author

September 02, 2014 05:44 PM

Phil,

You answered with "So you can't set a new render target and use the stencil buffer from a previous one.", and then asked exactly what I was trying to do. I was unable to come back to this for several days so I started a new thread, including code explaining exactly what I'm trying to do.

Maybe I misunderstand you answer, but I'm not setting a new render target and using the stencil buffer from a previous one. Are you saying that I need to set the rendertarget, write to the stencilbuffer, and then immediately send the rendertarget to the pixel shader for the comparison operation? This makes no sense.....you can't send an active rendertarget to the pixel shader, it will error out. Or do you mean that I can only perform a stencil test on the active rendertarget? In this case if I lose my stencil buffer as soon as I set the rendertarget, how can stencil buffering be in any way useful at all? Would I have to create the stencil and perform the comparison test all inside one draw call?

SO if what I'm trying to do isn't possible maybe I could trouble you to explain exactly how I can do this? Really all I'm trying to do is mask certain pixels out of a rendertarget with an early stencil test.....seems pretty simple. This is driving me nuts, and I've found several other examples of people with pretty much the same problem who haven't found a resolution, or people who made it work in XNA 3 but couldn't make it work in XNA 4. I found one guy who says you need to use preservecontents on a rendertarget in order to write to the stencil buffer, but still no dice, although my tests do indicate that the stencil buffer is never being written to.

I can't even find a decent explanation of how stencil buffering works. I might be conceptually completely out of the ballpark. For example, I *think* what I'm doing is comparing the stencil buffer of a rendertarget to a simple reference value. Is this correct? Am I comparing the contents of the rendertarget stencil buffer to the stencil buffer of the back buffer?, noting that I'm not drawing to the back buffer? Is drawing to the backbuffer the only circumstance where stencil buffering will work?

Or maybe you could help me with simple code describing how simple a stencil mask works, that actually does work?

Much Confusion. I really appreciate your help.

phil_t

8,087

September 03, 2014 05:46 AM

In this case if I lose my stencil buffer as soon as I set the rendertarget, how can stencil buffering be in any way useful at all? Would I have to create the stencil and perform the comparison test all inside one draw call?

Not inside one draw call, but in several draw calls without switching render targets. You generally draw stuff to set the stencil bits how you want (often disabling color writes), and then draw the "actual stuff" that draws based on the stencil comparison.

I think (not sure) that in raw DirectX in PCs you can manage the visual buffer and depth/stencil buffer separately, but they are tied together in XNA (presumably because of this limitation in the Xbox).

For example, I *think* what I'm doing is comparing the stencil buffer of a rendertarget to a simple reference value. Is this correct?

Yes. Or simply writing to the stencil buffer.

So for a very simple use case, maybe you want to draw a circle and have a square cut out of it. So you would first draw the square and have your DepthStencilState set up to set the stencil to 1*, say (GraphicsDevice.Clear will clear it to 0 by default, unless you specify otherwise). Since you don't want to actually see the square, you would also have disabled color writes by setting ColorWriteChannels to None in the BlendState you use. So now you have nothing visible, but you have a square in the stencil buffer where the values are 1.

Next, you would draw the circle (without changing rendertargets) with your DepthStencilState set up to only draw where there is a 1 in the stencil buffer. So the circle will draw everywhere except where the square is.

Here's an XNA stencil buffer example I found. I don't know if it will be helpful, but it should show you how to set up the DepthStencilState to get the functionality you want:

http://scott-franks.com/alpha-masking-in-xna-via-the-stencil-buffer/

*Note that when setting stencil values, the stencil is set depending on whether a pixel is output, even if that pixel is transparent. So if you are setting irregular shapes by drawing sprites, you generally want to be using alpha-testing (which discards transparent pixels), not alpha blending.

Cascade Quest - IceFall Games - Dev blog - twitter - youtube

danromeo

238

Author

September 04, 2014 12:31 AM

Phil,

Very helpful, thank you very much. Simply put, when I set a new rendertarget I lose the stencil buffers on previous rendertargets. Is this correct? So stenciling needs to be done in a multipass inside of a single SetRenderTarget, yes? What a PITA....

XNA Stencil Buffer problem, with code

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

XNA Stencil Buffer problem, with code

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines