It appears my question is too basic to be in the FAQ, so here it is..
In XNA, how do I "merge" the depth and stencil buffers? eg, so that a sprite appears to have a depth in the 3D scene, obscured by things on top?
The 3D depth buffer has a proper Z position, but the stencil buffer is just from 0 to 1. I've tried scaling the stencil Z position in to the 0 to 1 range, but this doesn't help.
The two buffers work as separate things... I want to copy the data from the depth buffer to make up a stencil buffer before drawing my 2d sprites... Surely there is a function already in place to do this? Several hours looking haven't found it yet.
Any nudge in the right direction is appriechiated.
Point sprites are a nice optimization when the hardware supports them, and when you can work within their limitations. The main benefit is that since you only need one vertex per sprite you can use a much smaller vertex buffer (with no index buffer necessary), which saves on memory, vertex processing load, and bandwidth. SpriteBatch just renders quads with a vertex buffer + index buffer, but is generally easier to use and gives you some more flexibility if you need it. Plus it doesn't require hardware support for point sprites, of course. If you're going to be rendering many many sprites (for something like a particle system), point sprites are a good choice if they're available.
Anyway your original question is unrelated to point sprites vs. SpriteBatch, it's simply an issue of using the depth buffer correctly. First of all, what you want is basic z-buffering behavior: you want to render all of your 3D geometry with z-writes and z-buffering enabled (RenderState.DepthBufferEnable = true and RenderState.DepthBufferWriteEnabled = true), and then render your sprites also with z-buffering enabled. As long as...
1. z-buffering is enabled 2. the depth buffer hasn't been cleared 3. you're setting a proper depth value for your sprites
...they should be tested for visibility just like 3D geometry. If you're not getting this behavior, I would suggest checking those 3 things.
Also on a side note, the stencil buffer doesn't hold "Z position"...it just holds an integer value. This value is typically 8-bits, can be 1-bit (or no bits) depending on the format of your DepthStencilBuffer. Either way it's just a value that's set depending on the values of RenderState.StencilFail and RenderState.StencilPass. If an incoming pixel passes the stencil test (specified by RenderState.StencilFunc) then StencilPass gets used, otherwise if it fails StencilFail gets used. At that point the stencil value doesn't get set to Z-position or anything like that...instead a simple operation is performed. It's either incremented, decremented, inverted, zeroed, or set to a value. See the documentation for StencilOperation for more details.
You also can't copy stencil data to Z or vice-versa. In fact you can't really directly manipulate the contents of the DepthStencilBuffer aside from clearing it...everything has to be done through render states and drawing geometry.
It all makes a lot more sense now I've slept on it. Point sprites is probably the way to go, what with the lower bandwidth. But..
Are you saying that I should have seen my non-point sprites working with the stencil buffer automatically, if all was well? I'd like to get my head around it, even if I end up using point sprites. So, as the depth buffer is modified by 3D objects, it also updates the stencil buffer when altered (at lower depth resolution depending on bits used), or do they only work as separate things? If it should work, how does the stencil buffer scale itself from the depth buffer?