# Managed DX Stencil Shadow Problems

## Recommended Posts

StoneDevil    122
private void ComputeContourEdges(SceneObject mesh, Light light)
{
m_contourEdges = new List<Edge>();

// put light into object position.
Matrix transform = Matrix.Transformation(new Vector3(0, 0, 0), new Quaternion(0, 0, 0, 0), m_shadowScale,

transform.Invert();
Vector3 lightPosition = light.GetParentNode().GetPosition();
lightPosition.TransformCoordinate(transform);

// grab the cached indices and vertices
CustomVertex.PositionOnly[] verts = mesh.GetCachedMeshVertices();
int[] indices = mesh.GetCachedMeshIndices();

Vector3 vert1, vert2, vert3;
Edge edge;
for (int i = 0; i < indices.Length; i += 3)
{
vert1 = verts[indices[i]].Position;
vert2 = verts[indices[i + 1]].Position;
vert3 = verts[indices[i + 2]].Position;

// should have 3 verts of the face
Vector3 p1 = vert1 - vert2;
Vector3 p2 = vert3 - vert2;

// normalise vectors
p1.Normalize();
p2.Normalize();

// normal = cross product of a and b
Vector3 normal = Vector3.Cross(p2, p1);
normal.Normalize();

// calculate the light direction vector with the face position and light position.
Vector3 facePos = vert1 + vert2 + vert3;
facePos.X /= 3.0f;
facePos.Y /= 3.0f;
facePos.Y /= 3.0f;

Vector3 lightDir = facePos - lightPosition;

// dot product light direction with normal if >= 0.0 add edges of the face that arent already there
if (Vector3.Dot(normal, lightDir) >= 0.0)
{
// for 3 edges add to list if they dont exist
edge.p1 = indices[i];
edge.p2 = indices[i + 1];

edge.p1 = indices[i + 1];
edge.p2 = indices[i + 2];

edge.p1 = indices[i + 2];
edge.p2 = indices[i];
}
else
{
}
}

ExtrudeContourEdges(verts, lightPosition, light.GetRange());

verts = null;
indices = null;
}

{
// swap edge to make first point < second
if (edge.p1 > edge.p2)
{
int temp = edge.p2;
edge.p2 = edge.p1;
edge.p1 = temp;
}

int index = CheckDuplicateEdge(edge);

if (index == -1) // add if no duplicate
else // remove the duplicate and dont add...edge is not contour if both polygons touching are in shadow
m_contourEdges.RemoveAt(index);
}

private int CheckDuplicateEdge(Edge edge)
{
//can change this method later to faster search method
for (int i = 0; i < m_contourEdges.Count; i++)
{
//if (edge.p1 > m_contourEdges[i].p1) // if its bigger than break...wont find a match
//    break;

if (!(edge.p1 != m_contourEdges[i].p1 || edge.p2 != m_contourEdges[i].p2)) // if edge matches
return i;
}

return -1;
}

private void ExtrudeContourEdges(CustomVertex.PositionOnly[] verts, Vector3 lightPosition, float lightRange)
{
m_shadowIndices = new int[m_contourEdges.Count * 6]; // 6 indices per edge
m_shadowVertices = new CustomVertex.PositionOnly[m_contourEdges.Count * 4]; // 4 vertices per edge

Vector3 vert1, vert2, vert3, vert4;
Vector3 lightDir;
int vertIndex = 0;
int index = 0;
for (int i = 0; i < m_contourEdges.Count; i++)
{
// set the 4 verts
vert1 = verts[m_contourEdges[i].p1].Position;
vert2 = verts[m_contourEdges[i].p2].Position;
vert3 = verts[m_contourEdges[i].p2].Position;
vert4 = verts[m_contourEdges[i].p1].Position;

// extrude the first 2 verts by direction
lightDir = vert1 - lightPosition;
lightDir.Normalize();
vert1 += (lightDir * lightRange);

lightDir = vert2 - lightPosition;
lightDir.Normalize();
vert2 += (lightDir * lightRange);

// map the indices

vertIndex += 4;
}
}


I run through all the faces in the mesh. dot product the normal to the light direction. And if its <= 0.0 then i add the edges the a list. but first check if there is a duplicate edge. if theres a duplicate edge i remove both edges because i only want the boundry edges. this is how i render the shadows:
Device device = DeviceManager.m_device;

// turn off all lights
for (int i = 0; i < device.Lights.Count; i++)
{
device.Lights[i].Enabled = false;
device.Lights[i].Update();
}

// render the scene with ambient light...dont draw to stencile
device.RenderState.StencilEnable = false;
device.RenderState.ZBufferEnable = true;
foreach (Node node in m_nodes)
{
node.Update();
}

// render all shadow volumes for each light
foreach (SceneObject obj in m_objects)
{
if (obj.GetType() != typeof(Light))
continue;

// cear the stencil buffer
device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);

// disable writing to zbuffer...color
device.RenderState.ZBufferWriteEnable = false;
device.RenderState.ColorWriteEnable = 0;

device.RenderState.StencilEnable = true;
device.RenderState.StencilFunction = Compare.Always;
device.RenderState.ReferenceStencil = 0x1;
device.RenderState.StencilZBufferFail = StencilOperation.Keep;
device.RenderState.StencilFail = StencilOperation.Keep;

device.RenderState.StencilPass = StencilOperation.Increment;

device.RenderState.CullMode = Cull.CounterClockwise;

device.RenderState.StencilPass = StencilOperation.Decrement;

device.RenderState.CullMode = Cull.Clockwise;

device.Lights[((Light)obj).GetIndex()].Enabled = true;
device.Lights[((Light)obj).GetIndex()].Update();

device.Clear(ClearFlags.ZBuffer, Color.Black, 1.0f, 0);
device.RenderState.ZBufferWriteEnable = true;
device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;
device.RenderState.CullMode = Cull.CounterClockwise;

// update all nodes
foreach (Node node in m_nodes)
{
node.Update();
}
}


the node.Update() method is the loop that renders all my objects in the scene. So i first turn off all the lights, render the whole scene with ambient light. The i loop through all my lights and render the shadow volumes to the stencil buffer. and then turn on that light and render the scene again. i think im missing something in the rendering now because no shadow is appearing. once i get the shadow appear i can post a screenshot of what it looks like. thanks for any help

##### Share on other sites
remigius    1172
It's been a while since I tinkered with shadow volumes, but as I recall a typical approach is to render the shadow volumes to the stencil buffer and then use a screensized quad with a transparent gray color to actually render the shadows over the scene based on your stencil. I'm not sure if this is "the only right way", but at first glance you seem to be missing that quad rendering step.

Perhaps my old shadow volume sample (6th on the list) may be of some help.

##### Share on other sites
StoneDevil    122
Alright so ive now written the code that caps my shadow volume which seems to be working well...so i have a fully capped shadow volume and ive change my rendering method and im getting a shadow but very broken...

the light is a point light above the teapot...

the new rendering code it

            Device device = DeviceManager.m_device;            // turn off all lights            for (int i = 0; i < device.Lights.Count; i++)            {                device.Lights[i].Enabled = false;                device.Lights[i].Update();            }            // render the scene with ambient light...dont draw to stencile            device.RenderState.StencilEnable = false;            foreach (Node node in m_nodes)            {                node.Update();            }            // render all shadow volumes for each light            foreach (SceneObject obj in m_objects)            {                if (obj.GetType() != typeof(Light))                    continue;                // cear the stencil buffer                device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);                // disable depth and color buffers and enable stencile                device.RenderState.ShadeMode = ShadeMode.Flat;                device.RenderState.ZBufferWriteEnable = false;                device.RenderState.ColorWriteEnable = 0;                device.RenderState.StencilEnable = true;                device.RenderState.ReferenceStencil = 0x1;                device.RenderState.StencilMask = 0xFFFFFF;                device.RenderState.StencilWriteMask = 0xFFFFFF;                                device.RenderState.TwoSidedStencilMode = true;                device.RenderState.StencilFunction = Compare.Always;                device.RenderState.StencilZBufferFail = StencilOperation.Keep;                device.RenderState.StencilFail = StencilOperation.Keep;                device.RenderState.StencilPass = StencilOperation.Increment;                device.RenderState.CounterClockwiseStencilFunction = Compare.Always;                device.RenderState.CounterClockwiseStencilZBufferFail = StencilOperation.Keep;                device.RenderState.CounterClockwiseStencilFail = StencilOperation.Keep;                device.RenderState.CounterClockwiseStencilPass = StencilOperation.Decrement;                                device.RenderState.CullMode = Cull.None;                ((Light)obj).UpdateShadowVolumes();                // set color and depth writing and enable the light                device.RenderState.ShadeMode = ShadeMode.Gouraud;                device.RenderState.CullMode = Cull.CounterClockwise;                device.RenderState.ZBufferWriteEnable = true;                device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;                device.RenderState.StencilEnable = true;                device.RenderState.ReferenceStencil = 0x1;                device.RenderState.StencilFunction = Compare.Greater;                device.RenderState.StencilPass = StencilOperation.Keep;                                device.Lights[((Light)obj).GetIndex()].Enabled = true;                device.Lights[((Light)obj).GetIndex()].Update();                // update all nodes                foreach (Node node in m_nodes)                {                    node.Update();                }            }

any ideas what im doing wrong.

##### Share on other sites
remigius    1172

Odd, particularly the 'stretching' to the right of the teapot. Have you tried rendering your actual shadow volumes to the scene, so you can check out if those look ok?

##### Share on other sites
StoneDevil    122
ive rendered the shadow volume to the scene and it seems fine....in the image you cant see the bottom cap but its there.

##### Share on other sites
remigius    1172

Hmm, the volumes do look correct indeed. I've been skimming over your code and I can't spot anything wrong at first glance. On the other hand, I'm not quite sure I understand how your code actually works [smile]

Exactly where are the volumes getting drawn to the stencil, does ((Light)obj).UpdateShadowVolumes(); take care of that?

Anyway, something to look at is the Device.Clear you only apply to the stencil before rendering the volumes. The ambient pass you render before that (with ZBufferWrites enabled afaik) should already fill up the zbuffer with depth data from the scene. Since you don't clear the zbuffer afterwards, the shadow volumes would be clipped by the zvalues of the ambient pass, which might account for the strange artifacts you're seeing.

It's a bit of a random guess, but you could try clearing both the stencil and zbuffer after the ambient pass. This should give you some overdraw, but it just might work.

##### Share on other sites
StoneDevil    122
hi,

thanks for you response....you were right that the shadow volumes do get rendered with ((Light)obj).UpdateShadowVolumes()...but i tried clearing the ZBuffer aswell as the stencil buffer before my shadow volume rendering and it didnt work properly.

also i ran my app in PIX for Windows and i was looking at the stencil buffer in there...you can assume what it would look like considering how the shadows turn out....and also i was interested why the depth buffer was all black....is this right?

[Edited by - StoneDevil on May 14, 2008 8:40:35 AM]

##### Share on other sites
StoneDevil    122
here is an image of what my stencil buffer looks like....its obviously not right...im not sure why though....the depth buffer is black....i dont know if thats the problem though...

##### Share on other sites
remigius    1172
Sorry, I got a bit absorbed in my own problems [smile]

I still don't have a clear cut answer for you, but the shadow volumes seem to be correct and the stencil doesn't, so you could narrow it down to something being off with the rendering to prepare the stencil. If you're sure about the renderstates, perhaps your shadow volumes are being clipped by the far clipping plane in your stencil pass? Reducing the distance they're extruded would help in this case (more info).

As for the black zbuffer, I just ran PIX with an app of mine and the color variation for most pixels in a scene is nearly invisible, unless you move something really close to the near plane. This seems expected behavior considering the distribution of zbuffer precission (more info), so I don't think it has anything to do with your problem.

##### Share on other sites
StoneDevil    122
hi,

thanks for the help and suggestions....i still cant get them perfect but i was able to improve the shadows by setting:

the shadows have improved but theres still a few problems...im not sure if this is because of the mesh im using. Maybe theres holes in the geometry. Im using the Mesh.Teapot and also tried with the airplane and ball that come with the sdk demos. the ball gives a complete shadow but the teapot and plane dont...i will post pictures....

also you can notice that the shadow volumes are effecting the acual mesh aswell..the teapot doesnt look proper...i am capping both ends of the shadow volume which is correct right?

##### Share on other sites
remigius    1172

The capping of both ends should be correct. With my zpass implementation I had to offset (move/translate) the volumes a small distance along the light vector, so the front caps were effectively covered by the visible geometry. From what I've read about zfail, this offset is also needed there. This could also be done with zbias, but that's tricky.

Looking at the pictures, there seems to be something strange going on when the 'light ray' intersects the mesh 4 times. This would also be consistent with the spere working well, since it would only pass through that twice. But I'm probably reading to much in these pictures, so I guess we're back to checking the renderstates and shadow volume again.

Each article I came across uses 0xffffffff for the stencil mask and write mask and from my understanding that would be the correct way. The 0x1 you're using might be cause for the latest shadow strangeness and could indicate something is going wrong with your previous approach when the stencil gets (or should get) values > 1. Reviewing your renderstates, I think they were correct, so if nothing is going wrong with the actual rendering, there might be something off with the volumes.

I really wish I could be more specific, but the best thing I can come up with is to have you check again if the shadow volumes aren't getting clipped by the far plane of your view frustum (more info, 2nd paragraph).

##### Share on other sites
StoneDevil    122
hi again,

ive been trying track down the problem with my renderstates...ive come across a problem though that i cant set my stencilmask or stencilwritemask to 0xFFFFFFFF.

will complain about cast to unsigned int. i can get it to set with

if i set the stencilmask and stencilwritemask to 0xFFFFFFFF i dont get anything drawn to my stencil buffer though. from what i can tell by debugging in PIX for windows.

could this be the problem. thanks again for the help.

##### Share on other sites
remigius    1172
It just might be... I'm a bit rusty, but I think you can tell C# to just trust your cast with this:

device.RenderState.StencilMask = unchecked((int)0xFFFFFFFF);

##### Share on other sites
StoneDevil    122
i think i finally got it....or atleast good enough for now...i had to change my shadow volume code because i was swapping edge points to check for duplicate edges faster, and now when im extuding the edges i am swappping them back to the original way....this fixed the problem....

thanks for all the help...a few pictures to show it working....