Managed DX Stencil Shadow Problems

Started by
12 comments, last by StoneDevil 15 years, 11 months ago
Hi, Im trying to get stencil shadows to work in managed directx. Im constructing the shadow volume for each object that each light lights in my scene. I cache my meshes vertices and indices and every frame i create a group of shadow volumes for each light in my scene. Currently im not capping my shadow volume but i plan to just use the faces that face the light as my cap. Im having some problems though and im not sure if its the rendering process or the construction of my shadow volume. Currently no shadow appears, ive redone my rendering code. But previously i had shadows rendering but not properly. Meaning holes in the shadow and shaded areas that shouldnt have been. I cunstruct my shadow volume like so.

private void ComputeContourEdges(SceneObject mesh, Light light)
        {
            m_contourEdges = new List<Edge>();

            // put light into object position.
            Matrix transform = Matrix.Transformation(new Vector3(0, 0, 0), new Quaternion(0, 0, 0, 0), m_shadowScale,
                                                    new Vector3(0, 0, 0), m_shadowOrientation, m_shadowPosition);

            transform.Invert();
            Vector3 lightPosition = light.GetParentNode().GetPosition();
            lightPosition.TransformCoordinate(transform);

            // grab the cached indices and vertices
            CustomVertex.PositionOnly[] verts = mesh.GetCachedMeshVertices();
            int[] indices = mesh.GetCachedMeshIndices();

            Vector3 vert1, vert2, vert3;
            Edge edge;
            for (int i = 0; i < indices.Length; i += 3)
            {
                // read three vert
                vert1 = verts[indices].Position;
                vert2 = verts[indices].Position;
                vert<span class="cpp-literal"><span class="cpp-number">3</span></span> = verts[indices].Position;

                <span class="cpp-comment">// should have 3 verts of the face</span>
                Vector<span class="cpp-literal"><span class="cpp-number">3</span></span> p<span class="cpp-literal"><span class="cpp-number">1</span></span> = vert<span class="cpp-literal"><span class="cpp-number">1</span></span> - vert<span class="cpp-literal"><span class="cpp-number">2</span></span>;
                Vector<span class="cpp-literal"><span class="cpp-number">3</span></span> p<span class="cpp-literal"><span class="cpp-number">2</span></span> = vert<span class="cpp-literal"><span class="cpp-number">3</span></span> - vert<span class="cpp-literal"><span class="cpp-number">2</span></span>;

                <span class="cpp-comment">// normalise vectors</span>
                p<span class="cpp-literal"><span class="cpp-number">1</span>.</span>Normalize();
                p<span class="cpp-literal"><span class="cpp-number">2</span>.</span>Normalize();

                <span class="cpp-comment">// normal = cross product of a and b</span>
                Vector<span class="cpp-literal"><span class="cpp-number">3</span></span> normal = Vector<span class="cpp-literal"><span class="cpp-number">3</span>.</span>Cross(p<span class="cpp-literal"><span class="cpp-number">2</span></span>, p<span class="cpp-literal"><span class="cpp-number">1</span></span>);
                normal.Normalize();

                <span class="cpp-comment">// calculate the light direction vector with the face position and light position.</span>
                Vector<span class="cpp-literal"><span class="cpp-number">3</span></span> facePos = vert<span class="cpp-literal"><span class="cpp-number">1</span></span> + vert<span class="cpp-literal"><span class="cpp-number">2</span></span> + vert<span class="cpp-literal"><span class="cpp-number">3</span></span>;
                facePos.X /= <span class="cpp-literal"><span class="cpp-number">3</span>.0f</span>;
                facePos.Y /= <span class="cpp-literal"><span class="cpp-number">3</span>.0f</span>;
                facePos.Y /= <span class="cpp-literal"><span class="cpp-number">3</span>.0f</span>;

                Vector<span class="cpp-literal"><span class="cpp-number">3</span></span> lightDir = facePos - lightPosition;

                <span class="cpp-comment">// dot product light direction with normal if &gt;= 0.0 add edges of the face that arent already there</span>
                <span class="cpp-keyword">if</span> (Vector<span class="cpp-literal"><span class="cpp-number">3</span>.D</span>ot(normal, lightDir) &gt;= <span class="cpp-literal"><span class="cpp-number">0</span>.<span class="cpp-number">0</span></span>)
                {
                    <span class="cpp-comment">// for 3 edges add to list if they dont exist</span>
                    edge.p<span class="cpp-literal"><span class="cpp-number">1</span></span> = indices<span style="font-weight:bold;">;
                    edge.p<span class="cpp-literal"><span class="cpp-number">2</span></span> = indices;
                    AddContourEdge(edge);

                    edge.p<span class="cpp-literal"><span class="cpp-number">1</span></span> = indices;
                    edge.p<span class="cpp-literal"><span class="cpp-number">2</span></span> = indices;
                    AddContourEdge(edge);

                    edge.p<span class="cpp-literal"><span class="cpp-number">1</span></span> = indices;
                    edge.p<span class="cpp-literal"><span class="cpp-number">2</span></span> = indices<span style="font-weight:bold;">;
                    AddContourEdge(edge);
                }
                <span class="cpp-keyword">else</span>
                {
                }
            }

            ExtrudeContourEdges(verts, lightPosition, light.GetRange());

            verts = <span class="cpp-literal">null</span>;
            indices = <span class="cpp-literal">null</span>;
        }

        <span class="cpp-keyword">private</span> <span class="cpp-keyword">void</span> AddContourEdge(Edge edge)
        {
            <span class="cpp-comment">// swap edge to make first point &lt; second</span>
            <span class="cpp-keyword">if</span> (edge.p<span class="cpp-literal"><span class="cpp-number">1</span></span> &gt; edge.p<span class="cpp-literal"><span class="cpp-number">2</span></span>)
            {
                <span class="cpp-keyword">int</span> temp = edge.p<span class="cpp-literal"><span class="cpp-number">2</span></span>;
                edge.p<span class="cpp-literal"><span class="cpp-number">2</span></span> = edge.p<span class="cpp-literal"><span class="cpp-number">1</span></span>;
                edge.p<span class="cpp-literal"><span class="cpp-number">1</span></span> = temp;
            }

            <span class="cpp-keyword">int</span> index = CheckDuplicateEdge(edge);

            <span class="cpp-keyword">if</span> (index == -<span class="cpp-literal"><span class="cpp-number">1</span></span>) <span class="cpp-comment">// add if no duplicate</span>
                m_contourEdges.Add(edge);
            <span class="cpp-keyword">else</span> <span class="cpp-comment">// remove the duplicate and dont add…edge is not contour if both polygons touching are in shadow</span>
                m_contourEdges.RemoveAt(index);
        }

        <span class="cpp-keyword">private</span> <span class="cpp-keyword">int</span> CheckDuplicateEdge(Edge edge)
        {
            <span class="cpp-comment">//can change this method later to faster search method</span>
            <span class="cpp-keyword">for</span> (<span class="cpp-keyword">int</span> i = <span class="cpp-literal"><span class="cpp-number">0</span></span>; i &lt; m_contourEdges.Count; i++)
            {
                <span class="cpp-comment">//if (edge.p1 &gt; m_contourEdges<span style="font-weight:bold;">.p1) // if its bigger than break…wont find a match</span>
                <span class="cpp-comment">//    break;</span>

                <span class="cpp-keyword">if</span> (!(edge.p<span class="cpp-literal"><span class="cpp-number">1</span></span> != m_contourEdges<span style="font-weight:bold;">.p<span class="cpp-literal"><span class="cpp-number">1</span></span> || edge.p<span class="cpp-literal"><span class="cpp-number">2</span></span> != m_contourEdges<span style="font-weight:bold;">.p<span class="cpp-literal"><span class="cpp-number">2</span></span>)) <span class="cpp-comment">// if edge matches</span>
                    <span class="cpp-keyword">return</span> i;
            }

            <span class="cpp-keyword">return</span> -<span class="cpp-literal"><span class="cpp-number">1</span></span>;
        }

        <span class="cpp-keyword">private</span> <span class="cpp-keyword">void</span> ExtrudeContourEdges(CustomVertex.PositionOnly[] verts, Vector<span class="cpp-literal"><span class="cpp-number">3</span></span> lightPosition, <span class="cpp-keyword">float</span> lightRange)
        {
            m_shadowIndices = <span class="vb-function">new</span> <span class="cpp-keyword">int</span>[m_contourEdges.Count * <span class="cpp-literal"><span class="cpp-number">6</span></span>]; <span class="cpp-comment">// 6 indices per edge</span>
            m_shadowVertices = <span class="vb-function">new</span> CustomVertex.PositionOnly[m_contourEdges.Count * <span class="cpp-literal"><span class="cpp-number">4</span></span>]; <span class="cpp-comment">// 4 vertices per edge</span>

            Vector<span class="cpp-literal"><span class="cpp-number">3</span></span> vert<span class="cpp-literal"><span class="cpp-number">1</span></span>, vert<span class="cpp-literal"><span class="cpp-number">2</span></span>, vert<span class="cpp-literal"><span class="cpp-number">3</span></span>, vert<span class="cpp-literal"><span class="cpp-number">4</span></span>;
            Vector<span class="cpp-literal"><span class="cpp-number">3</span></span> lightDir;
            <span class="cpp-keyword">int</span> vertIndex = <span class="cpp-literal"><span class="cpp-number">0</span></span>;
            <span class="cpp-keyword">int</span> index = <span class="cpp-literal"><span class="cpp-number">0</span></span>;
            <span class="cpp-keyword">for</span> (<span class="cpp-keyword">int</span> i = <span class="cpp-literal"><span class="cpp-number">0</span></span>; i &lt; m_contourEdges.Count; i++)
            {
                <span class="cpp-comment">// set the 4 verts</span>
                vert<span class="cpp-literal"><span class="cpp-number">1</span></span> = verts[m_contourEdges<span style="font-weight:bold;">.p<span class="cpp-literal"><span class="cpp-number">1</span></span>].Position;
                vert<span class="cpp-literal"><span class="cpp-number">2</span></span> = verts[m_contourEdges<span style="font-weight:bold;">.p<span class="cpp-literal"><span class="cpp-number">2</span></span>].Position;
                vert<span class="cpp-literal"><span class="cpp-number">3</span></span> = verts[m_contourEdges<span style="font-weight:bold;">.p<span class="cpp-literal"><span class="cpp-number">2</span></span>].Position;
                vert<span class="cpp-literal"><span class="cpp-number">4</span></span> = verts[m_contourEdges<span style="font-weight:bold;">.p<span class="cpp-literal"><span class="cpp-number">1</span></span>].Position;

                <span class="cpp-comment">// extrude the first 2 verts by direction</span>
                lightDir = vert<span class="cpp-literal"><span class="cpp-number">1</span></span> - lightPosition;
                lightDir.Normalize();
                vert<span class="cpp-literal"><span class="cpp-number">1</span></span> += (lightDir * lightRange);

                lightDir = vert<span class="cpp-literal"><span class="cpp-number">2</span></span> - lightPosition;
                lightDir.Normalize();
                vert<span class="cpp-literal"><span class="cpp-number">2</span></span> += (lightDir * lightRange);

                m_shadowVertices[vertIndex].Position = vert<span class="cpp-literal"><span class="cpp-number">1</span></span>;
                m_shadowVertices[vertIndex + <span class="cpp-literal"><span class="cpp-number">1</span></span>].Position = vert<span class="cpp-literal"><span class="cpp-number">2</span></span>;
                m_shadowVertices[vertIndex + <span class="cpp-literal"><span class="cpp-number">2</span></span>].Position = vert<span class="cpp-literal"><span class="cpp-number">3</span></span>;
                m_shadowVertices[vertIndex + <span class="cpp-literal"><span class="cpp-number">3</span></span>].Position = vert<span class="cpp-literal"><span class="cpp-number">4</span></span>;

                <span class="cpp-comment">// map the indices</span>
                m_shadowIndices[index++] = vertIndex;
                m_shadowIndices[index++] = vertIndex + <span class="cpp-literal"><span class="cpp-number">1</span></span>;
                m_shadowIndices[index++] = vertIndex + <span class="cpp-literal"><span class="cpp-number">2</span></span>;
                m_shadowIndices[index++] = vertIndex;
                m_shadowIndices[index++] = vertIndex + <span class="cpp-literal"><span class="cpp-number">2</span></span>;
                m_shadowIndices[index++] = vertIndex + <span class="cpp-literal"><span class="cpp-number">3</span></span>;

                vertIndex += <span class="cpp-literal"><span class="cpp-number">4</span></span>;
            }
        }

</pre></div><!–ENDSCRIPT–>

I run through all the faces in the mesh.  dot product the normal to the light direction.  And if its &lt;= 0.0 then i add the edges the a list. but first check if there is a duplicate edge.  if theres a duplicate edge i remove both edges because i &#111;nly want the boundry edges.

this is how i render the shadows:

<!–STARTSCRIPT–><!–source lang="c#"–><div class="source"><pre>
Device device = DeviceManager.m_device;

            <span class="cpp-comment">// turn off all lights</span>
            <span class="cpp-keyword">for</span> (<span class="cpp-keyword">int</span> i = <span class="cpp-literal"><span class="cpp-number">0</span></span>; i &lt; device.Lights.Count; i++)
            {
                device.Lights<span style="font-weight:bold;">.Enabled = <span class="cpp-literal">false</span>;
                device.Lights<span style="font-weight:bold;">.Update();
            }

            <span class="cpp-comment">// render the scene with ambient light…dont draw to stencile</span>
            device.RenderState.StencilEnable = <span class="cpp-literal">false</span>;
            device.RenderState.ZBufferEnable = <span class="cpp-literal">true</span>;
            <span class="cpp-keyword">foreach</span> (Node node <span class="cpp-keyword">in</span> m_nodes)
            {
                node.Update();
            }

            <span class="cpp-comment">// render all shadow volumes for each light</span>
            <span class="cpp-keyword">foreach</span> (SceneObject obj <span class="cpp-keyword">in</span> m_objects)
            {
                <span class="cpp-keyword">if</span> (obj.GetType() != <span class="vb-function">typeof</span>(Light))
                    <span class="cpp-keyword">continue</span>;

                <span class="cpp-comment">// cear the stencil buffer</span>
                device.Clear(ClearFlags.Stencil, Color.Black, <span class="cpp-literal"><span class="cpp-number">1</span>.0f</span>, <span class="cpp-literal"><span class="cpp-number">0</span></span>);

                <span class="cpp-comment">// disable writing to zbuffer…color</span>
                device.RenderState.ZBufferWriteEnable = <span class="cpp-literal">false</span>;
                device.RenderState.ColorWriteEnable = <span class="cpp-literal"><span class="cpp-number">0</span></span>;

                device.RenderState.StencilEnable = <span class="cpp-literal">true</span>;
                device.RenderState.StencilFunction = Compare.Always;
                device.RenderState.ReferenceStencil = <span class="cpp-literal">0x1</span>;
                device.RenderState.StencilMask = <span class="cpp-literal">0xFFFFFF</span>;
                device.RenderState.StencilWriteMask = <span class="cpp-literal">0xFFFFFF</span>;
                device.RenderState.StencilZBufferFail = StencilOperation.Keep;
                device.RenderState.StencilFail = StencilOperation.Keep;

                device.RenderState.StencilPass = StencilOperation.Increment;

                device.RenderState.CullMode = Cull.CounterClockwise;
                ((Light)obj).UpdateShadowVolumes();

                device.RenderState.StencilPass = StencilOperation.Decrement;

                device.RenderState.CullMode = Cull.Clockwise;
                ((Light)obj).UpdateShadowVolumes();
                
                device.Lights[((Light)obj).GetIndex()].Enabled = <span class="cpp-literal">true</span>;
                device.Lights[((Light)obj).GetIndex()].Update();

                device.Clear(ClearFlags.ZBuffer, Color.Black, <span class="cpp-literal"><span class="cpp-number">1</span>.0f</span>, <span class="cpp-literal"><span class="cpp-number">0</span></span>);
                device.RenderState.ZBufferWriteEnable = <span class="cpp-literal">true</span>;
                device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;
                device.RenderState.CullMode = Cull.CounterClockwise;

                <span class="cpp-comment">// update all nodes</span>
                <span class="cpp-keyword">foreach</span> (Node node <span class="cpp-keyword">in</span> m_nodes)
                {
                    node.Update();
                }
            }

</pre></div><!–ENDSCRIPT–>

the node.Update() method is the loop that renders all my objects in the scene.  So i first turn off all the lights, render the whole scene with ambient light.  The i loop through all my lights and render the shadow volumes to the stencil buffer.  and then turn &#111;n that light and render the scene again.

i think im missing something in the rendering now because no shadow is appearing. &#111;nce i get the shadow appear i can post a screenshot of what it looks like.

thanks for any help

Advertisement
It's been a while since I tinkered with shadow volumes, but as I recall a typical approach is to render the shadow volumes to the stencil buffer and then use a screensized quad with a transparent gray color to actually render the shadows over the scene based on your stencil. I'm not sure if this is "the only right way", but at first glance you seem to be missing that quad rendering step.

Perhaps my old shadow volume sample (6th on the list) may be of some help.
Rim van Wersch [ MDXInfo ] [ XNAInfo ] [ YouTube ] - Do yourself a favor and bookmark this excellent free online D3D/shader book!
Alright so ive now written the code that caps my shadow volume which seems to be working well...so i have a fully capped shadow volume and ive change my rendering method and im getting a shadow but very broken...



the light is a point light above the teapot...

the new rendering code it

            Device device = DeviceManager.m_device;            // turn off all lights            for (int i = 0; i < device.Lights.Count; i++)            {                device.Lights.Enabled = false;                device.Lights.Update();            }            // render the scene with ambient light...dont draw to stencile            device.RenderState.StencilEnable = false;            foreach (Node node in m_nodes)            {                node.Update();            }            // render all shadow volumes for each light            foreach (SceneObject obj in m_objects)            {                if (obj.GetType() != typeof(Light))                    continue;                // cear the stencil buffer                device.Clear(ClearFlags.Stencil, Color.Black, 1.0f, 0);                // disable depth and color buffers and enable stencile                device.RenderState.ShadeMode = ShadeMode.Flat;                device.RenderState.ZBufferWriteEnable = false;                device.RenderState.ColorWriteEnable = 0;                device.RenderState.StencilEnable = true;                device.RenderState.ReferenceStencil = 0x1;                device.RenderState.StencilMask = 0xFFFFFF;                device.RenderState.StencilWriteMask = 0xFFFFFF;                                device.RenderState.TwoSidedStencilMode = true;                device.RenderState.StencilFunction = Compare.Always;                device.RenderState.StencilZBufferFail = StencilOperation.Keep;                device.RenderState.StencilFail = StencilOperation.Keep;                device.RenderState.StencilPass = StencilOperation.Increment;                device.RenderState.CounterClockwiseStencilFunction = Compare.Always;                device.RenderState.CounterClockwiseStencilZBufferFail = StencilOperation.Keep;                device.RenderState.CounterClockwiseStencilFail = StencilOperation.Keep;                device.RenderState.CounterClockwiseStencilPass = StencilOperation.Decrement;                                device.RenderState.CullMode = Cull.None;                ((Light)obj).UpdateShadowVolumes();                // set color and depth writing and enable the light                device.RenderState.ShadeMode = ShadeMode.Gouraud;                device.RenderState.CullMode = Cull.CounterClockwise;                device.RenderState.ZBufferWriteEnable = true;                device.RenderState.ColorWriteEnable = ColorWriteEnable.RedGreenBlueAlpha;                device.RenderState.StencilEnable = true;                device.RenderState.ReferenceStencil = 0x1;                device.RenderState.StencilFunction = Compare.Greater;                device.RenderState.StencilPass = StencilOperation.Keep;                                device.Lights[((Light)obj).GetIndex()].Enabled = true;                device.Lights[((Light)obj).GetIndex()].Update();                // update all nodes                foreach (Node node in m_nodes)                {                    node.Update();                }            }


any ideas what im doing wrong.

Odd, particularly the 'stretching' to the right of the teapot. Have you tried rendering your actual shadow volumes to the scene, so you can check out if those look ok?

Rim van Wersch [ MDXInfo ] [ XNAInfo ] [ YouTube ] - Do yourself a favor and bookmark this excellent free online D3D/shader book!
ive rendered the shadow volume to the scene and it seems fine....in the image you cant see the bottom cap but its there.

Free Image Hosting at www.ImageShack.us

Hmm, the volumes do look correct indeed. I've been skimming over your code and I can't spot anything wrong at first glance. On the other hand, I'm not quite sure I understand how your code actually works [smile]

Exactly where are the volumes getting drawn to the stencil, does ((Light)obj).UpdateShadowVolumes(); take care of that?

Anyway, something to look at is the Device.Clear you only apply to the stencil before rendering the volumes. The ambient pass you render before that (with ZBufferWrites enabled afaik) should already fill up the zbuffer with depth data from the scene. Since you don't clear the zbuffer afterwards, the shadow volumes would be clipped by the zvalues of the ambient pass, which might account for the strange artifacts you're seeing.

It's a bit of a random guess, but you could try clearing both the stencil and zbuffer after the ambient pass. This should give you some overdraw, but it just might work.
Rim van Wersch [ MDXInfo ] [ XNAInfo ] [ YouTube ] - Do yourself a favor and bookmark this excellent free online D3D/shader book!
hi,

thanks for you response....you were right that the shadow volumes do get rendered with ((Light)obj).UpdateShadowVolumes()...but i tried clearing the ZBuffer aswell as the stencil buffer before my shadow volume rendering and it didnt work properly.

Free Image Hosting at www.ImageShack.us

also i ran my app in PIX for Windows and i was looking at the stencil buffer in there...you can assume what it would look like considering how the shadows turn out....and also i was interested why the depth buffer was all black....is this right?

[Edited by - StoneDevil on May 14, 2008 8:40:35 AM]
here is an image of what my stencil buffer looks like....its obviously not right...im not sure why though....the depth buffer is black....i dont know if thats the problem though...

Free Image Hosting at www.ImageShack.us
Sorry, I got a bit absorbed in my own problems [smile]

I still don't have a clear cut answer for you, but the shadow volumes seem to be correct and the stencil doesn't, so you could narrow it down to something being off with the rendering to prepare the stencil. If you're sure about the renderstates, perhaps your shadow volumes are being clipped by the far clipping plane in your stencil pass? Reducing the distance they're extruded would help in this case (more info).

As for the black zbuffer, I just ran PIX with an app of mine and the color variation for most pixels in a scene is nearly invisible, unless you move something really close to the near plane. This seems expected behavior considering the distribution of zbuffer precission (more info), so I don't think it has anything to do with your problem.
Rim van Wersch [ MDXInfo ] [ XNAInfo ] [ YouTube ] - Do yourself a favor and bookmark this excellent free online D3D/shader book!
hi,

thanks for the help and suggestions....i still cant get them perfect but i was able to improve the shadows by setting:

device.RenderState.StencilMask = 0x1;
device.RenderState.StencilWriteMask = 0x1;

the shadows have improved but theres still a few problems...im not sure if this is because of the mesh im using. Maybe theres holes in the geometry. Im using the Mesh.Teapot and also tried with the airplane and ball that come with the sdk demos. the ball gives a complete shadow but the teapot and plane dont...i will post pictures....

also you can notice that the shadow volumes are effecting the acual mesh aswell..the teapot doesnt look proper...i am capping both ends of the shadow volume which is correct right?

Free Image Hosting at www.ImageShack.usFree Image Hosting at www.ImageShack.us

This topic is closed to new replies.

Advertisement