Hi there,

I have a 2D scene with three layers - background, center and foreground. I'm rendering the central layer first so I can perform lighting calculations, and then want to render the background followed by the foreground.

I'm currently trying to set up a stencil buffer when rendering the central layer, so any objects drawn in the background pass will skip pixels that already have central there.

Only trouble is the stencil test doesn't seem to apply (unless I'm completely missing how it's supposed to work), the background objects always overwrite the foreground objects (as they're second).

I'm setting everything up like this :

myColourTarget = new RenderTarget2D( aGraphicsDevice, width, height, false, SurfaceFormat.Color, DepthFormat.Depth24Stencil8 );

...

graphicsDevice.SetRenderTarget( myColourTarget );
graphicsDevice.Clear( ClearOptions.Stencil | ClearOptions.Target, ClearColor, -1.0f, 0 );

myDepthStencilState.StencilEnable = true;
myDepthStencilState.StencilFunction = CompareFunction.Always;
myDepthStencilState.ReferenceStencil = 1;
myDepthStencilState.StencilPass = StencilOperation.Replace;
myDepthStencilState.DepthBufferEnable = false;

spriteBatch.Begin( SpriteSortMode.Immediate, BlendState.AlphaBlend, null, myDepthStencilState, null, objectEffect);

// draw all central objects

spriteBatch.End()

myDepthStencilState.StencilEnable = true;
myDepthStencilState.StencilFunction = CompareFunction.Equal;
myDepthStencilState.ReferenceStencil = 0;
myDepthStencilState.StencilPass = StencilOperation.Keep;
myDepthStencilState.DepthBufferEnable = false;

spriteBatch.Begin( SpriteSortMode.Immediate, BlendState.AlphaBlend, null, myDepthStencilState, null, objectEffect);

// draw all background objects

spriteBatch.End();

I've tried to do the same sort of effect using depth testing, but I can't seem to get the alpha blending correct - pixels from central objects with an alpha value of 0 always overwrite background images, and the clear colour for the render target bleeds through ( http://i.imgur.com/lRkh3Mj.png )

Any ideas what I'm doing wrong with the stencil test... I figured that the first stencil state would write a 1 whenever a pixel value is written, and the second state would set things up sot that it will only keep the pixel if there's a 0 in the stencil buffer.

Cheers!

Found the issue - modifying the parameters of a single DepthStencilState doesn't seem to work. My renderer now uses two separate DepthStencilState instances and everything works as expected.

The reason why that didn't work for you was because state objects in D3D10+ are immutable, once created with their initial states they can't be modified, so you must either destroy and recreate the object with new states (not recommended as runtime destruction/creation is expensive) or create multiple state objects. This can seem like a bit of a pain-in-the-ass when you first encounter it, but in theory it should give you higher performance as (1) a whole bunch of state can be set in a single API call, and (2) hardware groups state internally in a similar manner anyway (in practice the high optimization of D3D9 drivers over more than a decade, coupled with lazy state changes in the driver, means that you may not actually get that increased performance - oh well).

Yeah that makes sense - thanks! It's always good to know the why of things :)