Jump to content
  • Advertisement
Sign in to follow this  
StoneDevil

Managed DX Stencil Shadow Problems

This topic is 3707 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi, Im trying to get stencil shadows to work in managed directx. Im constructing the shadow volume for each object that each light lights in my scene. I cache my meshes vertices and indices and every frame i create a group of shadow volumes for each light in my scene. Currently im not capping my shadow volume but i plan to just use the faces that face the light as my cap. Im having some problems though and im not sure if its the rendering process or the construction of my shadow volume. Currently no shadow appears, ive redone my rendering code. But previously i had shadows rendering but not properly. Meaning holes in the shadow and shaded areas that shouldnt have been. I cunstruct my shadow volume like so.
private void ComputeContourEdges(SceneObject mesh, Light light)
        {
            m_contourEdges = new List<Edge>();

            // put light into object position.
            Matrix transform = Matrix.Transformation(new Vector3(0, 0, 0), new Quaternion(0, 0, 0, 0), m_shadowScale,
                                                    new Vector3(0, 0, 0), m_shadowOrientation, m_shadowPosition);

            transform.Invert();
            Vector3 lightPosition = light.GetParentNode().GetPosition();
            lightPosition.TransformCoordinate(transform);

            // grab the cached indices and vertices
            CustomVertex.PositionOnly[] verts = mesh.GetCachedMeshVertices();
            int[] indices = mesh.GetCachedMeshIndices();

            Vector3 vert1, vert2, vert3;
            Edge edge;
            for (int i = 0; i < indices.Length; i += 3)
            {
                // read three vert
                vert1 = verts[indices].Position;
                vert2 = verts[indices[i + 1]].Position;
                vert3 = verts[indices[i + 2]].Position;

                // should have 3 verts of the face
                Vector3 p1 = vert1 - vert2;
                Vector3 p2 = vert3 - vert2;

                // normalise vectors
                p1.Normalize();
                p2.Normalize();

                // normal = cross product of a and b
                Vector3 normal = Vector3.Cross(p2, p1);
                normal.Normalize();

                // calculate the light direction vector with the face position and light position.
                Vector3 facePos = vert1 + vert2 + vert3;
                facePos.X /= 3.0f;
                facePos.Y /= 3.0f;
                facePos.Y /= 3.0f;

                Vector3 lightDir = facePos - lightPosition;

                // dot product light direction with normal if >= 0.0 add edges of the face that arent already there
                if (Vector3.Dot(normal, lightDir) >= 0.0)
                {
                    // for 3 edges add to list if they dont exist
                    edge.p1 = indices;
                    edge.p2 = indices[i + 1];
                    AddContourEdge(edge);

                    edge.p1 = indices[i + 1];
                    edge.p2 = indices[i + 2];
                    AddContourEdge(edge);

                    edge.p1 = indices[i + 2];
                    edge.p2 = indices;
                    AddContourEdge(edge);
                }
                else
                {
                }
            }

            ExtrudeContourEdges(verts, lightPosition, light.GetRange());

            verts = null;
            indices = null;
        }

        private void AddContourEdge(Edge edge)
        {
            // swap edge to make first point < second
            if (edge.p1 > edge.p2)
            {
                int temp = edge.p2;
                edge.p2 = edge.p1;
                edge.p1 = temp;
            }

            int index = CheckDuplicateEdge(edge);

            if (index == -1) // add if no duplicate
                m_contourEdges.Add(edge);
            else // remove the duplicate and dont add...edge is not contour if both polygons touching are in shadow
                m_contourEdges.RemoveAt(index);
        }

        private int CheckDuplicateEdge(Edge edge)
        {
            //can change this method later to faster search method
            for (int i = 0; i < m_contourEdges.Count; i++)
            {
                //if (edge.p1 > m_contourEdges.p1) // if its bigger than break...wont find a match
                //    break;

                if (!(edge.p1 != m_contourEdges.p1 || edge.p2 != m_contourEdges.p2)) // if edge matches
                    return i;
            }

            return -1;
        }

        private void ExtrudeContourEdges(CustomVertex.PositionOnly[] verts, Vector3 lightPosition, float lightRange)
        {
            m_shadowIndices = new int[m_contourEdges.Count * 6]; // 6 indices per edge
            m_shadowVertices = new CustomVertex.PositionOnly[m_contourEdges.Count * 4]; // 4 vertices per edge

            Vector3 vert1, vert2, vert3, vert4;
            Vector3 lightDir;
            int vertIndex = 0;
            int index = 0;
            for (int i = 0; i < m_contourEdges.Count; i++)
            {
                // set the 4 verts
                vert1 = verts[m_contourEdges.p1].Position;
                vert2 = verts[m_contourEdges.p2].Position;
                vert3 = verts[m_contourEdges.p2].Position;
                vert4 = verts[m_contourEdges.p1].Position;

                // extrude the first 2 verts by direction
                lightDir = vert1 - lightPosition;
                lightDir.Normalize();
                vert1 += (lightDir * lightRange);

                lightDir = vert2 - lightPosition;
                lightDir.Normalize();
                vert2 += (lightDir * lightRange);

                m_shadowVertices[vertIndex].Position = vert1;
                m_shadowVertices[vertIndex + 1].Position = vert2;
                m_shadowVertices[vertIndex + 2].Position = vert3;
                m_shadowVertices[vertIndex + 3].Position = vert4;

                // map the indices
                m_shadowIndices[index++] = vertIndex;
                m_shadowIndices[index++] = vertIndex + 1;
                m_shadowIndices[index++] = vertIndex + 2;
                m_shadowIndices[index++] = vertIndex;
                m_shadowIndices[index++] = vertIndex + 2;
                m_shadowIndices[index++] = vertIndex + 3;

                vertIndex += 4;
            }
        }

I run through all the faces in the mesh. dot product the normal to the light direction. And if its <= 0.0 then i add the edges the a list. but first check if there is a duplicate edge. if theres a duplicate edge i remove both edges because i only want the boundry edges. this is how i render the shadows:
Device device = DeviceManager.m_device;

            // turn off all lights
            for (int i = 0; i < device.Lights.Count; i++)
            {
                device.Lights.Enabled = false;
                device.Lights.Update();
            }

            // render the scene with ambient light...dont draw to stencile
            device.RenderState.StencilEnable = false;
            device.RenderState.ZBufferEnable = true;
            foreach (Node node in m_nodes)
            {
                node.Update();
            }

            // render all shadow volumes for each light
            foreach (SceneObject obj in m_objects)
            {
                if (obj.GetType() != typeof(Light))
                    continue;

                // cear the stencil buffer
                device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);

                // disable writing to zbuffer...color
                device.RenderState.ZBufferWriteEnable = false;
                device.RenderState.ColorWriteEnable = 0;

                device.RenderState.StencilEnable = true;
                device.RenderState.StencilFunction = Compare.Always;
                device.RenderState.ReferenceStencil = 0x1;
                device.RenderState.StencilMask = 0xFFFFFF;
                device.RenderState.StencilWriteMask = 0xFFFFFF;
                device.RenderState.StencilZBufferFail = StencilOperation.Keep;
                device.RenderState.StencilFail = StencilOperation.Keep;

                device.RenderState.StencilPass = StencilOperation.Increment;

                device.RenderState.CullMode = Cull.CounterClockwise;
                ((Light)obj).UpdateShadowVolumes();

                device.RenderState.StencilPass = StencilOperation.Decrement;

                device.RenderState.CullMode = Cull.Clockwise;
                ((Light)obj).UpdateShadowVolumes();
                
                device.Lights[((Light)obj).GetIndex()].Enabled = true;
                device.Lights[((Light)obj).GetIndex()].Update();

                device.Clear(ClearFlags.ZBuffer, Color.Black, 1.0f, 0);
                device.RenderState.ZBufferWriteEnable = true;
                device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;
                device.RenderState.CullMode = Cull.CounterClockwise;

                // update all nodes
                foreach (Node node in m_nodes)
                {
                    node.Update();
                }
            }

the node.Update() method is the loop that renders all my objects in the scene. So i first turn off all the lights, render the whole scene with ambient light. The i loop through all my lights and render the shadow volumes to the stencil buffer. and then turn on that light and render the scene again. i think im missing something in the rendering now because no shadow is appearing. once i get the shadow appear i can post a screenshot of what it looks like. thanks for any help

Share this post


Link to post
Share on other sites
Advertisement
It's been a while since I tinkered with shadow volumes, but as I recall a typical approach is to render the shadow volumes to the stencil buffer and then use a screensized quad with a transparent gray color to actually render the shadows over the scene based on your stencil. I'm not sure if this is "the only right way", but at first glance you seem to be missing that quad rendering step.

Perhaps my old shadow volume sample (6th on the list) may be of some help.

Share this post


Link to post
Share on other sites
Alright so ive now written the code that caps my shadow volume which seems to be working well...so i have a fully capped shadow volume and ive change my rendering method and im getting a shadow but very broken...



the light is a point light above the teapot...

the new rendering code it


Device device = DeviceManager.m_device;

// turn off all lights
for (int i = 0; i < device.Lights.Count; i++)
{
device.Lights.Enabled = false;
device.Lights.Update();
}

// render the scene with ambient light...dont draw to stencile
device.RenderState.StencilEnable = false;
foreach (Node node in m_nodes)
{
node.Update();
}

// render all shadow volumes for each light
foreach (SceneObject obj in m_objects)
{
if (obj.GetType() != typeof(Light))
continue;

// cear the stencil buffer
device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);

// disable depth and color buffers and enable stencile
device.RenderState.ShadeMode = ShadeMode.Flat;
device.RenderState.ZBufferWriteEnable = false;
device.RenderState.ColorWriteEnable = 0;
device.RenderState.StencilEnable = true;
device.RenderState.ReferenceStencil = 0x1;
device.RenderState.StencilMask = 0xFFFFFF;
device.RenderState.StencilWriteMask = 0xFFFFFF;

device.RenderState.TwoSidedStencilMode = true;

device.RenderState.StencilFunction = Compare.Always;
device.RenderState.StencilZBufferFail = StencilOperation.Keep;
device.RenderState.StencilFail = StencilOperation.Keep;
device.RenderState.StencilPass = StencilOperation.Increment;

device.RenderState.CounterClockwiseStencilFunction = Compare.Always;
device.RenderState.CounterClockwiseStencilZBufferFail = StencilOperation.Keep;
device.RenderState.CounterClockwiseStencilFail = StencilOperation.Keep;
device.RenderState.CounterClockwiseStencilPass = StencilOperation.Decrement;

device.RenderState.CullMode = Cull.None;
((Light)obj).UpdateShadowVolumes();

// set color and depth writing and enable the light
device.RenderState.ShadeMode = ShadeMode.Gouraud;
device.RenderState.CullMode = Cull.CounterClockwise;
device.RenderState.ZBufferWriteEnable = true;
device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;
device.RenderState.StencilEnable = true;
device.RenderState.ReferenceStencil = 0x1;
device.RenderState.StencilFunction = Compare.Greater;
device.RenderState.StencilPass = StencilOperation.Keep;

device.Lights[((Light)obj).GetIndex()].Enabled = true;
device.Lights[((Light)obj).GetIndex()].Update();

// update all nodes
foreach (Node node in m_nodes)
{
node.Update();
}
}



any ideas what im doing wrong.

Share this post


Link to post
Share on other sites

Odd, particularly the 'stretching' to the right of the teapot. Have you tried rendering your actual shadow volumes to the scene, so you can check out if those look ok?

Share this post


Link to post
Share on other sites
ive rendered the shadow volume to the scene and it seems fine....in the image you cant see the bottom cap but its there.

Free Image Hosting at www.ImageShack.us

Share this post


Link to post
Share on other sites

Hmm, the volumes do look correct indeed. I've been skimming over your code and I can't spot anything wrong at first glance. On the other hand, I'm not quite sure I understand how your code actually works [smile]

Exactly where are the volumes getting drawn to the stencil, does ((Light)obj).UpdateShadowVolumes(); take care of that?

Anyway, something to look at is the Device.Clear you only apply to the stencil before rendering the volumes. The ambient pass you render before that (with ZBufferWrites enabled afaik) should already fill up the zbuffer with depth data from the scene. Since you don't clear the zbuffer afterwards, the shadow volumes would be clipped by the zvalues of the ambient pass, which might account for the strange artifacts you're seeing.

It's a bit of a random guess, but you could try clearing both the stencil and zbuffer after the ambient pass. This should give you some overdraw, but it just might work.

Share this post


Link to post
Share on other sites
hi,

thanks for you response....you were right that the shadow volumes do get rendered with ((Light)obj).UpdateShadowVolumes()...but i tried clearing the ZBuffer aswell as the stencil buffer before my shadow volume rendering and it didnt work properly.

Free Image Hosting at www.ImageShack.us

also i ran my app in PIX for Windows and i was looking at the stencil buffer in there...you can assume what it would look like considering how the shadows turn out....and also i was interested why the depth buffer was all black....is this right?

[Edited by - StoneDevil on May 14, 2008 8:40:35 AM]

Share this post


Link to post
Share on other sites
here is an image of what my stencil buffer looks like....its obviously not right...im not sure why though....the depth buffer is black....i dont know if thats the problem though...

Free Image Hosting at www.ImageShack.us

Share this post


Link to post
Share on other sites
Sorry, I got a bit absorbed in my own problems [smile]

I still don't have a clear cut answer for you, but the shadow volumes seem to be correct and the stencil doesn't, so you could narrow it down to something being off with the rendering to prepare the stencil. If you're sure about the renderstates, perhaps your shadow volumes are being clipped by the far clipping plane in your stencil pass? Reducing the distance they're extruded would help in this case (more info).

As for the black zbuffer, I just ran PIX with an app of mine and the color variation for most pixels in a scene is nearly invisible, unless you move something really close to the near plane. This seems expected behavior considering the distribution of zbuffer precission (more info), so I don't think it has anything to do with your problem.

Share this post


Link to post
Share on other sites
hi,

thanks for the help and suggestions....i still cant get them perfect but i was able to improve the shadows by setting:

device.RenderState.StencilMask = 0x1;
device.RenderState.StencilWriteMask = 0x1;

the shadows have improved but theres still a few problems...im not sure if this is because of the mesh im using. Maybe theres holes in the geometry. Im using the Mesh.Teapot and also tried with the airplane and ball that come with the sdk demos. the ball gives a complete shadow but the teapot and plane dont...i will post pictures....

also you can notice that the shadow volumes are effecting the acual mesh aswell..the teapot doesnt look proper...i am capping both ends of the shadow volume which is correct right?

Free Image Hosting at www.ImageShack.usFree Image Hosting at www.ImageShack.us

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!