Sign in to follow this  
StoneDevil

Managed DX Stencil Shadow Problems

Recommended Posts

StoneDevil    122
Hi, Im trying to get stencil shadows to work in managed directx. Im constructing the shadow volume for each object that each light lights in my scene. I cache my meshes vertices and indices and every frame i create a group of shadow volumes for each light in my scene. Currently im not capping my shadow volume but i plan to just use the faces that face the light as my cap. Im having some problems though and im not sure if its the rendering process or the construction of my shadow volume. Currently no shadow appears, ive redone my rendering code. But previously i had shadows rendering but not properly. Meaning holes in the shadow and shaded areas that shouldnt have been. I cunstruct my shadow volume like so.
private void ComputeContourEdges(SceneObject mesh, Light light)
        {
            m_contourEdges = new List<Edge>();

            // put light into object position.
            Matrix transform = Matrix.Transformation(new Vector3(0, 0, 0), new Quaternion(0, 0, 0, 0), m_shadowScale,
                                                    new Vector3(0, 0, 0), m_shadowOrientation, m_shadowPosition);

            transform.Invert();
            Vector3 lightPosition = light.GetParentNode().GetPosition();
            lightPosition.TransformCoordinate(transform);

            // grab the cached indices and vertices
            CustomVertex.PositionOnly[] verts = mesh.GetCachedMeshVertices();
            int[] indices = mesh.GetCachedMeshIndices();

            Vector3 vert1, vert2, vert3;
            Edge edge;
            for (int i = 0; i < indices.Length; i += 3)
            {
                // read three vert
                vert1 = verts[indices[i]].Position;
                vert2 = verts[indices[i + 1]].Position;
                vert3 = verts[indices[i + 2]].Position;

                // should have 3 verts of the face
                Vector3 p1 = vert1 - vert2;
                Vector3 p2 = vert3 - vert2;

                // normalise vectors
                p1.Normalize();
                p2.Normalize();

                // normal = cross product of a and b
                Vector3 normal = Vector3.Cross(p2, p1);
                normal.Normalize();

                // calculate the light direction vector with the face position and light position.
                Vector3 facePos = vert1 + vert2 + vert3;
                facePos.X /= 3.0f;
                facePos.Y /= 3.0f;
                facePos.Y /= 3.0f;

                Vector3 lightDir = facePos - lightPosition;

                // dot product light direction with normal if >= 0.0 add edges of the face that arent already there
                if (Vector3.Dot(normal, lightDir) >= 0.0)
                {
                    // for 3 edges add to list if they dont exist
                    edge.p1 = indices[i];
                    edge.p2 = indices[i + 1];
                    AddContourEdge(edge);

                    edge.p1 = indices[i + 1];
                    edge.p2 = indices[i + 2];
                    AddContourEdge(edge);

                    edge.p1 = indices[i + 2];
                    edge.p2 = indices[i];
                    AddContourEdge(edge);
                }
                else
                {
                }
            }

            ExtrudeContourEdges(verts, lightPosition, light.GetRange());

            verts = null;
            indices = null;
        }

        private void AddContourEdge(Edge edge)
        {
            // swap edge to make first point < second
            if (edge.p1 > edge.p2)
            {
                int temp = edge.p2;
                edge.p2 = edge.p1;
                edge.p1 = temp;
            }

            int index = CheckDuplicateEdge(edge);

            if (index == -1) // add if no duplicate
                m_contourEdges.Add(edge);
            else // remove the duplicate and dont add...edge is not contour if both polygons touching are in shadow
                m_contourEdges.RemoveAt(index);
        }

        private int CheckDuplicateEdge(Edge edge)
        {
            //can change this method later to faster search method
            for (int i = 0; i < m_contourEdges.Count; i++)
            {
                //if (edge.p1 > m_contourEdges[i].p1) // if its bigger than break...wont find a match
                //    break;

                if (!(edge.p1 != m_contourEdges[i].p1 || edge.p2 != m_contourEdges[i].p2)) // if edge matches
                    return i;
            }

            return -1;
        }

        private void ExtrudeContourEdges(CustomVertex.PositionOnly[] verts, Vector3 lightPosition, float lightRange)
        {
            m_shadowIndices = new int[m_contourEdges.Count * 6]; // 6 indices per edge
            m_shadowVertices = new CustomVertex.PositionOnly[m_contourEdges.Count * 4]; // 4 vertices per edge

            Vector3 vert1, vert2, vert3, vert4;
            Vector3 lightDir;
            int vertIndex = 0;
            int index = 0;
            for (int i = 0; i < m_contourEdges.Count; i++)
            {
                // set the 4 verts
                vert1 = verts[m_contourEdges[i].p1].Position;
                vert2 = verts[m_contourEdges[i].p2].Position;
                vert3 = verts[m_contourEdges[i].p2].Position;
                vert4 = verts[m_contourEdges[i].p1].Position;

                // extrude the first 2 verts by direction
                lightDir = vert1 - lightPosition;
                lightDir.Normalize();
                vert1 += (lightDir * lightRange);

                lightDir = vert2 - lightPosition;
                lightDir.Normalize();
                vert2 += (lightDir * lightRange);

                m_shadowVertices[vertIndex].Position = vert1;
                m_shadowVertices[vertIndex + 1].Position = vert2;
                m_shadowVertices[vertIndex + 2].Position = vert3;
                m_shadowVertices[vertIndex + 3].Position = vert4;

                // map the indices
                m_shadowIndices[index++] = vertIndex;
                m_shadowIndices[index++] = vertIndex + 1;
                m_shadowIndices[index++] = vertIndex + 2;
                m_shadowIndices[index++] = vertIndex;
                m_shadowIndices[index++] = vertIndex + 2;
                m_shadowIndices[index++] = vertIndex + 3;

                vertIndex += 4;
            }
        }

I run through all the faces in the mesh. dot product the normal to the light direction. And if its <= 0.0 then i add the edges the a list. but first check if there is a duplicate edge. if theres a duplicate edge i remove both edges because i only want the boundry edges. this is how i render the shadows:
Device device = DeviceManager.m_device;

            // turn off all lights
            for (int i = 0; i < device.Lights.Count; i++)
            {
                device.Lights[i].Enabled = false;
                device.Lights[i].Update();
            }

            // render the scene with ambient light...dont draw to stencile
            device.RenderState.StencilEnable = false;
            device.RenderState.ZBufferEnable = true;
            foreach (Node node in m_nodes)
            {
                node.Update();
            }

            // render all shadow volumes for each light
            foreach (SceneObject obj in m_objects)
            {
                if (obj.GetType() != typeof(Light))
                    continue;

                // cear the stencil buffer
                device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);

                // disable writing to zbuffer...color
                device.RenderState.ZBufferWriteEnable = false;
                device.RenderState.ColorWriteEnable = 0;

                device.RenderState.StencilEnable = true;
                device.RenderState.StencilFunction = Compare.Always;
                device.RenderState.ReferenceStencil = 0x1;
                device.RenderState.StencilMask = 0xFFFFFF;
                device.RenderState.StencilWriteMask = 0xFFFFFF;
                device.RenderState.StencilZBufferFail = StencilOperation.Keep;
                device.RenderState.StencilFail = StencilOperation.Keep;

                device.RenderState.StencilPass = StencilOperation.Increment;

                device.RenderState.CullMode = Cull.CounterClockwise;
                ((Light)obj).UpdateShadowVolumes();

                device.RenderState.StencilPass = StencilOperation.Decrement;

                device.RenderState.CullMode = Cull.Clockwise;
                ((Light)obj).UpdateShadowVolumes();
                
                device.Lights[((Light)obj).GetIndex()].Enabled = true;
                device.Lights[((Light)obj).GetIndex()].Update();

                device.Clear(ClearFlags.ZBuffer, Color.Black, 1.0f, 0);
                device.RenderState.ZBufferWriteEnable = true;
                device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;
                device.RenderState.CullMode = Cull.CounterClockwise;

                // update all nodes
                foreach (Node node in m_nodes)
                {
                    node.Update();
                }
            }

the node.Update() method is the loop that renders all my objects in the scene. So i first turn off all the lights, render the whole scene with ambient light. The i loop through all my lights and render the shadow volumes to the stencil buffer. and then turn on that light and render the scene again. i think im missing something in the rendering now because no shadow is appearing. once i get the shadow appear i can post a screenshot of what it looks like. thanks for any help

Share this post


Link to post
Share on other sites
remigius    1172
It's been a while since I tinkered with shadow volumes, but as I recall a typical approach is to render the shadow volumes to the stencil buffer and then use a screensized quad with a transparent gray color to actually render the shadows over the scene based on your stencil. I'm not sure if this is "the only right way", but at first glance you seem to be missing that quad rendering step.

Perhaps my old shadow volume sample (6th on the list) may be of some help.

Share this post


Link to post
Share on other sites
StoneDevil    122
Alright so ive now written the code that caps my shadow volume which seems to be working well...so i have a fully capped shadow volume and ive change my rendering method and im getting a shadow but very broken...



the light is a point light above the teapot...

the new rendering code it


Device device = DeviceManager.m_device;

// turn off all lights
for (int i = 0; i < device.Lights.Count; i++)
{
device.Lights[i].Enabled = false;
device.Lights[i].Update();
}

// render the scene with ambient light...dont draw to stencile
device.RenderState.StencilEnable = false;
foreach (Node node in m_nodes)
{
node.Update();
}

// render all shadow volumes for each light
foreach (SceneObject obj in m_objects)
{
if (obj.GetType() != typeof(Light))
continue;

// cear the stencil buffer
device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);

// disable depth and color buffers and enable stencile
device.RenderState.ShadeMode = ShadeMode.Flat;
device.RenderState.ZBufferWriteEnable = false;
device.RenderState.ColorWriteEnable = 0;
device.RenderState.StencilEnable = true;
device.RenderState.ReferenceStencil = 0x1;
device.RenderState.StencilMask = 0xFFFFFF;
device.RenderState.StencilWriteMask = 0xFFFFFF;

device.RenderState.TwoSidedStencilMode = true;

device.RenderState.StencilFunction = Compare.Always;
device.RenderState.StencilZBufferFail = StencilOperation.Keep;
device.RenderState.StencilFail = StencilOperation.Keep;
device.RenderState.StencilPass = StencilOperation.Increment;

device.RenderState.CounterClockwiseStencilFunction = Compare.Always;
device.RenderState.CounterClockwiseStencilZBufferFail = StencilOperation.Keep;
device.RenderState.CounterClockwiseStencilFail = StencilOperation.Keep;
device.RenderState.CounterClockwiseStencilPass = StencilOperation.Decrement;

device.RenderState.CullMode = Cull.None;
((Light)obj).UpdateShadowVolumes();

// set color and depth writing and enable the light
device.RenderState.ShadeMode = ShadeMode.Gouraud;
device.RenderState.CullMode = Cull.CounterClockwise;
device.RenderState.ZBufferWriteEnable = true;
device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;
device.RenderState.StencilEnable = true;
device.RenderState.ReferenceStencil = 0x1;
device.RenderState.StencilFunction = Compare.Greater;
device.RenderState.StencilPass = StencilOperation.Keep;

device.Lights[((Light)obj).GetIndex()].Enabled = true;
device.Lights[((Light)obj).GetIndex()].Update();

// update all nodes
foreach (Node node in m_nodes)
{
node.Update();
}
}



any ideas what im doing wrong.

Share this post


Link to post
Share on other sites
remigius    1172

Odd, particularly the 'stretching' to the right of the teapot. Have you tried rendering your actual shadow volumes to the scene, so you can check out if those look ok?

Share this post


Link to post
Share on other sites
remigius    1172

Hmm, the volumes do look correct indeed. I've been skimming over your code and I can't spot anything wrong at first glance. On the other hand, I'm not quite sure I understand how your code actually works [smile]

Exactly where are the volumes getting drawn to the stencil, does ((Light)obj).UpdateShadowVolumes(); take care of that?

Anyway, something to look at is the Device.Clear you only apply to the stencil before rendering the volumes. The ambient pass you render before that (with ZBufferWrites enabled afaik) should already fill up the zbuffer with depth data from the scene. Since you don't clear the zbuffer afterwards, the shadow volumes would be clipped by the zvalues of the ambient pass, which might account for the strange artifacts you're seeing.

It's a bit of a random guess, but you could try clearing both the stencil and zbuffer after the ambient pass. This should give you some overdraw, but it just might work.

Share this post


Link to post
Share on other sites
StoneDevil    122
hi,

thanks for you response....you were right that the shadow volumes do get rendered with ((Light)obj).UpdateShadowVolumes()...but i tried clearing the ZBuffer aswell as the stencil buffer before my shadow volume rendering and it didnt work properly.

Free Image Hosting at www.ImageShack.us

also i ran my app in PIX for Windows and i was looking at the stencil buffer in there...you can assume what it would look like considering how the shadows turn out....and also i was interested why the depth buffer was all black....is this right?

[Edited by - StoneDevil on May 14, 2008 8:40:35 AM]

Share this post


Link to post
Share on other sites
StoneDevil    122
here is an image of what my stencil buffer looks like....its obviously not right...im not sure why though....the depth buffer is black....i dont know if thats the problem though...

Free Image Hosting at www.ImageShack.us

Share this post


Link to post
Share on other sites
remigius    1172
Sorry, I got a bit absorbed in my own problems [smile]

I still don't have a clear cut answer for you, but the shadow volumes seem to be correct and the stencil doesn't, so you could narrow it down to something being off with the rendering to prepare the stencil. If you're sure about the renderstates, perhaps your shadow volumes are being clipped by the far clipping plane in your stencil pass? Reducing the distance they're extruded would help in this case (more info).

As for the black zbuffer, I just ran PIX with an app of mine and the color variation for most pixels in a scene is nearly invisible, unless you move something really close to the near plane. This seems expected behavior considering the distribution of zbuffer precission (more info), so I don't think it has anything to do with your problem.

Share this post


Link to post
Share on other sites
StoneDevil    122
hi,

thanks for the help and suggestions....i still cant get them perfect but i was able to improve the shadows by setting:

device.RenderState.StencilMask = 0x1;
device.RenderState.StencilWriteMask = 0x1;

the shadows have improved but theres still a few problems...im not sure if this is because of the mesh im using. Maybe theres holes in the geometry. Im using the Mesh.Teapot and also tried with the airplane and ball that come with the sdk demos. the ball gives a complete shadow but the teapot and plane dont...i will post pictures....

also you can notice that the shadow volumes are effecting the acual mesh aswell..the teapot doesnt look proper...i am capping both ends of the shadow volume which is correct right?

Free Image Hosting at www.ImageShack.usFree Image Hosting at www.ImageShack.us

Share this post


Link to post
Share on other sites
remigius    1172

The capping of both ends should be correct. With my zpass implementation I had to offset (move/translate) the volumes a small distance along the light vector, so the front caps were effectively covered by the visible geometry. From what I've read about zfail, this offset is also needed there. This could also be done with zbias, but that's tricky.

Looking at the pictures, there seems to be something strange going on when the 'light ray' intersects the mesh 4 times. This would also be consistent with the spere working well, since it would only pass through that twice. But I'm probably reading to much in these pictures, so I guess we're back to checking the renderstates and shadow volume again.

Each article I came across uses 0xffffffff for the stencil mask and write mask and from my understanding that would be the correct way. The 0x1 you're using might be cause for the latest shadow strangeness and could indicate something is going wrong with your previous approach when the stencil gets (or should get) values > 1. Reviewing your renderstates, I think they were correct, so if nothing is going wrong with the actual rendering, there might be something off with the volumes.

I really wish I could be more specific, but the best thing I can come up with is to have you check again if the shadow volumes aren't getting clipped by the far plane of your view frustum (more info, 2nd paragraph).

Share this post


Link to post
Share on other sites
StoneDevil    122
hi again,

ive been trying track down the problem with my renderstates...ive come across a problem though that i cant set my stencilmask or stencilwritemask to 0xFFFFFFFF.

device.RenderState.StencilMask = 0xFFFFFFFF

will complain about cast to unsigned int. i can get it to set with

device.SetRenderState(RenderState.StencilMask, 0xFFFFFFFF)

if i set the stencilmask and stencilwritemask to 0xFFFFFFFF i dont get anything drawn to my stencil buffer though. from what i can tell by debugging in PIX for windows.

could this be the problem. thanks again for the help.

Share this post


Link to post
Share on other sites
remigius    1172
It just might be... I'm a bit rusty, but I think you can tell C# to just trust your cast with this:


device.RenderState.StencilMask = unchecked((int)0xFFFFFFFF);

Share this post


Link to post
Share on other sites
StoneDevil    122
i think i finally got it....or atleast good enough for now...i had to change my shadow volume code because i was swapping edge points to check for duplicate edges faster, and now when im extuding the edges i am swappping them back to the original way....this fixed the problem....

Free Image Hosting at www.ImageShack.usFree Image Hosting at www.ImageShack.us

thanks for all the help...a few pictures to show it working....

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this