# Managed DX Stencil Shadow Problems

This topic is 3707 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

private void ComputeContourEdges(SceneObject mesh, Light light)
{
m_contourEdges = new List<Edge>();

// put light into object position.
Matrix transform = Matrix.Transformation(new Vector3(0, 0, 0), new Quaternion(0, 0, 0, 0), m_shadowScale,

transform.Invert();
Vector3 lightPosition = light.GetParentNode().GetPosition();
lightPosition.TransformCoordinate(transform);

// grab the cached indices and vertices
CustomVertex.PositionOnly[] verts = mesh.GetCachedMeshVertices();
int[] indices = mesh.GetCachedMeshIndices();

Vector3 vert1, vert2, vert3;
Edge edge;
for (int i = 0; i < indices.Length; i += 3)
{
vert1 = verts[indices].Position;
vert2 = verts[indices[i + 1]].Position;
vert3 = verts[indices[i + 2]].Position;

// should have 3 verts of the face
Vector3 p1 = vert1 - vert2;
Vector3 p2 = vert3 - vert2;

// normalise vectors
p1.Normalize();
p2.Normalize();

// normal = cross product of a and b
Vector3 normal = Vector3.Cross(p2, p1);
normal.Normalize();

// calculate the light direction vector with the face position and light position.
Vector3 facePos = vert1 + vert2 + vert3;
facePos.X /= 3.0f;
facePos.Y /= 3.0f;
facePos.Y /= 3.0f;

Vector3 lightDir = facePos - lightPosition;

// dot product light direction with normal if >= 0.0 add edges of the face that arent already there
if (Vector3.Dot(normal, lightDir) >= 0.0)
{
// for 3 edges add to list if they dont exist
edge.p1 = indices;
edge.p2 = indices[i + 1];

edge.p1 = indices[i + 1];
edge.p2 = indices[i + 2];

edge.p1 = indices[i + 2];
edge.p2 = indices;
}
else
{
}
}

ExtrudeContourEdges(verts, lightPosition, light.GetRange());

verts = null;
indices = null;
}

{
// swap edge to make first point < second
if (edge.p1 > edge.p2)
{
int temp = edge.p2;
edge.p2 = edge.p1;
edge.p1 = temp;
}

int index = CheckDuplicateEdge(edge);

if (index == -1) // add if no duplicate
else // remove the duplicate and dont add...edge is not contour if both polygons touching are in shadow
m_contourEdges.RemoveAt(index);
}

private int CheckDuplicateEdge(Edge edge)
{
//can change this method later to faster search method
for (int i = 0; i < m_contourEdges.Count; i++)
{
//if (edge.p1 > m_contourEdges.p1) // if its bigger than break...wont find a match
//    break;

if (!(edge.p1 != m_contourEdges.p1 || edge.p2 != m_contourEdges.p2)) // if edge matches
return i;
}

return -1;
}

private void ExtrudeContourEdges(CustomVertex.PositionOnly[] verts, Vector3 lightPosition, float lightRange)
{
m_shadowIndices = new int[m_contourEdges.Count * 6]; // 6 indices per edge
m_shadowVertices = new CustomVertex.PositionOnly[m_contourEdges.Count * 4]; // 4 vertices per edge

Vector3 vert1, vert2, vert3, vert4;
Vector3 lightDir;
int vertIndex = 0;
int index = 0;
for (int i = 0; i < m_contourEdges.Count; i++)
{
// set the 4 verts
vert1 = verts[m_contourEdges.p1].Position;
vert2 = verts[m_contourEdges.p2].Position;
vert3 = verts[m_contourEdges.p2].Position;
vert4 = verts[m_contourEdges.p1].Position;

// extrude the first 2 verts by direction
lightDir = vert1 - lightPosition;
lightDir.Normalize();
vert1 += (lightDir * lightRange);

lightDir = vert2 - lightPosition;
lightDir.Normalize();
vert2 += (lightDir * lightRange);

// map the indices

vertIndex += 4;
}
}

I run through all the faces in the mesh. dot product the normal to the light direction. And if its <= 0.0 then i add the edges the a list. but first check if there is a duplicate edge. if theres a duplicate edge i remove both edges because i only want the boundry edges. this is how i render the shadows:
Device device = DeviceManager.m_device;

// turn off all lights
for (int i = 0; i < device.Lights.Count; i++)
{
device.Lights.Enabled = false;
device.Lights.Update();
}

// render the scene with ambient light...dont draw to stencile
device.RenderState.StencilEnable = false;
device.RenderState.ZBufferEnable = true;
foreach (Node node in m_nodes)
{
node.Update();
}

// render all shadow volumes for each light
foreach (SceneObject obj in m_objects)
{
if (obj.GetType() != typeof(Light))
continue;

// cear the stencil buffer
device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);

// disable writing to zbuffer...color
device.RenderState.ZBufferWriteEnable = false;
device.RenderState.ColorWriteEnable = 0;

device.RenderState.StencilEnable = true;
device.RenderState.StencilFunction = Compare.Always;
device.RenderState.ReferenceStencil = 0x1;
device.RenderState.StencilZBufferFail = StencilOperation.Keep;
device.RenderState.StencilFail = StencilOperation.Keep;

device.RenderState.StencilPass = StencilOperation.Increment;

device.RenderState.CullMode = Cull.CounterClockwise;

device.RenderState.StencilPass = StencilOperation.Decrement;

device.RenderState.CullMode = Cull.Clockwise;

device.Lights[((Light)obj).GetIndex()].Enabled = true;
device.Lights[((Light)obj).GetIndex()].Update();

device.Clear(ClearFlags.ZBuffer, Color.Black, 1.0f, 0);
device.RenderState.ZBufferWriteEnable = true;
device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;
device.RenderState.CullMode = Cull.CounterClockwise;

// update all nodes
foreach (Node node in m_nodes)
{
node.Update();
}
}

the node.Update() method is the loop that renders all my objects in the scene. So i first turn off all the lights, render the whole scene with ambient light. The i loop through all my lights and render the shadow volumes to the stencil buffer. and then turn on that light and render the scene again. i think im missing something in the rendering now because no shadow is appearing. once i get the shadow appear i can post a screenshot of what it looks like. thanks for any help

##### Share on other sites
It's been a while since I tinkered with shadow volumes, but as I recall a typical approach is to render the shadow volumes to the stencil buffer and then use a screensized quad with a transparent gray color to actually render the shadows over the scene based on your stencil. I'm not sure if this is "the only right way", but at first glance you seem to be missing that quad rendering step.

Perhaps my old shadow volume sample (6th on the list) may be of some help.

##### Share on other sites
Alright so ive now written the code that caps my shadow volume which seems to be working well...so i have a fully capped shadow volume and ive change my rendering method and im getting a shadow but very broken...

the light is a point light above the teapot...

the new rendering code it

Device device = DeviceManager.m_device;

// turn off all lights
for (int i = 0; i < device.Lights.Count; i++)
{
device.Lights.Enabled = false;
device.Lights.Update();
}

// render the scene with ambient light...dont draw to stencile
device.RenderState.StencilEnable = false;
foreach (Node node in m_nodes)
{
node.Update();
}

// render all shadow volumes for each light
foreach (SceneObject obj in m_objects)
{
if (obj.GetType() != typeof(Light))
continue;

// cear the stencil buffer
device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);

// disable depth and color buffers and enable stencile
device.RenderState.ZBufferWriteEnable = false;
device.RenderState.ColorWriteEnable = 0;
device.RenderState.StencilEnable = true;
device.RenderState.ReferenceStencil = 0x1;

device.RenderState.TwoSidedStencilMode = true;

device.RenderState.StencilFunction = Compare.Always;
device.RenderState.StencilZBufferFail = StencilOperation.Keep;
device.RenderState.StencilFail = StencilOperation.Keep;
device.RenderState.StencilPass = StencilOperation.Increment;

device.RenderState.CounterClockwiseStencilFunction = Compare.Always;
device.RenderState.CounterClockwiseStencilZBufferFail = StencilOperation.Keep;
device.RenderState.CounterClockwiseStencilFail = StencilOperation.Keep;
device.RenderState.CounterClockwiseStencilPass = StencilOperation.Decrement;

device.RenderState.CullMode = Cull.None;

// set color and depth writing and enable the light
device.RenderState.CullMode = Cull.CounterClockwise;
device.RenderState.ZBufferWriteEnable = true;
device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;
device.RenderState.StencilEnable = true;
device.RenderState.ReferenceStencil = 0x1;
device.RenderState.StencilFunction = Compare.Greater;
device.RenderState.StencilPass = StencilOperation.Keep;

device.Lights[((Light)obj).GetIndex()].Enabled = true;
device.Lights[((Light)obj).GetIndex()].Update();

// update all nodes
foreach (Node node in m_nodes)
{
node.Update();
}
}

any ideas what im doing wrong.

##### Share on other sites

Odd, particularly the 'stretching' to the right of the teapot. Have you tried rendering your actual shadow volumes to the scene, so you can check out if those look ok?

##### Share on other sites
ive rendered the shadow volume to the scene and it seems fine....in the image you cant see the bottom cap but its there.

##### Share on other sites

Hmm, the volumes do look correct indeed. I've been skimming over your code and I can't spot anything wrong at first glance. On the other hand, I'm not quite sure I understand how your code actually works [smile]

Exactly where are the volumes getting drawn to the stencil, does ((Light)obj).UpdateShadowVolumes(); take care of that?

Anyway, something to look at is the Device.Clear you only apply to the stencil before rendering the volumes. The ambient pass you render before that (with ZBufferWrites enabled afaik) should already fill up the zbuffer with depth data from the scene. Since you don't clear the zbuffer afterwards, the shadow volumes would be clipped by the zvalues of the ambient pass, which might account for the strange artifacts you're seeing.

It's a bit of a random guess, but you could try clearing both the stencil and zbuffer after the ambient pass. This should give you some overdraw, but it just might work.

##### Share on other sites
hi,

thanks for you response....you were right that the shadow volumes do get rendered with ((Light)obj).UpdateShadowVolumes()...but i tried clearing the ZBuffer aswell as the stencil buffer before my shadow volume rendering and it didnt work properly.

also i ran my app in PIX for Windows and i was looking at the stencil buffer in there...you can assume what it would look like considering how the shadows turn out....and also i was interested why the depth buffer was all black....is this right?

[Edited by - StoneDevil on May 14, 2008 8:40:35 AM]

##### Share on other sites
here is an image of what my stencil buffer looks like....its obviously not right...im not sure why though....the depth buffer is black....i dont know if thats the problem though...

##### Share on other sites
Sorry, I got a bit absorbed in my own problems [smile]

I still don't have a clear cut answer for you, but the shadow volumes seem to be correct and the stencil doesn't, so you could narrow it down to something being off with the rendering to prepare the stencil. If you're sure about the renderstates, perhaps your shadow volumes are being clipped by the far clipping plane in your stencil pass? Reducing the distance they're extruded would help in this case (more info).

As for the black zbuffer, I just ran PIX with an app of mine and the color variation for most pixels in a scene is nearly invisible, unless you move something really close to the near plane. This seems expected behavior considering the distribution of zbuffer precission (more info), so I don't think it has anything to do with your problem.

##### Share on other sites
hi,

thanks for the help and suggestions....i still cant get them perfect but i was able to improve the shadows by setting:

the shadows have improved but theres still a few problems...im not sure if this is because of the mesh im using. Maybe theres holes in the geometry. Im using the Mesh.Teapot and also tried with the airplane and ball that come with the sdk demos. the ball gives a complete shadow but the teapot and plane dont...i will post pictures....

also you can notice that the shadow volumes are effecting the acual mesh aswell..the teapot doesnt look proper...i am capping both ends of the shadow volume which is correct right?

1. 1
2. 2
Rutin
21
3. 3
JoeJ
18
4. 4
5. 5

• 14
• 40
• 23
• 13
• 13
• ### Forum Statistics

• Total Topics
631722
• Total Posts
3001892
×