Stencil Shadow Problem Please Look (Screenie)

Started by
12 comments, last by BlackMoons 19 years, 6 months ago
Please look at this: http://www.webjeff.com/img/newshadow.jpg I have no idea how to fix it, I can't figure this out. Has anyone else seen anything like this before or why it may happen like that??? Thanks Jeff.
Advertisement
how do you determine the winding of verts for the extruded faces?
I bet it doesn't happen if you use a utility sphere like one created from D3DXCreateSphere. Check to see if this is realy the case. If so most likely your models aren't closed well. They might look ok but the triangles that make up the model don't really touch each other. Just fix the model.
Don't shoot! I'm with the science team.....
Are you using the normals and the dot product to only extrude triangles whos front faces are facing the light?

How far are you extruding the volumes? I had a problem like this and I figured the bottom cap of the shadow volume must of been going past the far clip plane, try lowering the extrusion distance to just more than will ever be needed.
I just go through my indicies starting with 0.


Here is my ShadowVolume Code:

#include "stdafx.h"#include "ShadowVolume.h"//-----------------------------------------------------------------------------// Name: render// Desc://-----------------------------------------------------------------------------HRESULT ShadowVolume::render( LPDIRECT3DDEVICE9 pd3dDevice ){    pd3dDevice->SetFVF( D3DFVF_XYZ );    return pd3dDevice->DrawPrimitiveUP( D3DPT_TRIANGLELIST, m_dwNumVertices/3,                                        m_pVertices, sizeof(D3DXVECTOR3) );}//-----------------------------------------------------------------------------// Name: buildShadowVolume()// Desc: Takes a mesh as input, and uses it to build a shadow volume. The//       technique used considers each triangle of the mesh, and adds it's//       edges to a temporary list. The edge list is maintained, such that//       only silohuette edges are kept. Finally, the silohuette edges are//       extruded to make the shadow volume vertex list.//-----------------------------------------------------------------------------void ShadowVolume::CheckDual(LPDIRECT3DDEVICE9 pd3dDevice){	D3DCAPS9 d3dCaps;	pd3dDevice->GetDeviceCaps( &d3dCaps );// D3DADAPTER_DEFAULT, D3DDEVTYPE_HAL,	if( ( d3dCaps.StencilCaps & D3DSTENCILCAPS_TWOSIDED ) != 0 )		g_bTwoSidedStencilsAvailable = true;}HRESULT ShadowVolume::buildShadowVolume( void* pVert,void* pInd,DWORD numFaces,DWORD numVerts,D3DXVECTOR3 vLight,D3DXMATRIX mat ){	struct MeshVertex { 		D3DXVECTOR3 p, n;		FLOAT tu,tv;};    MeshVertex *pVertices = (MeshVertex*)pVert;    WORD       *pIndices  = (WORD*)pInd;    DWORD dwNumFaces    = numFaces;    DWORD dwNumVertices = numVerts;    WORD* pEdges = new WORD[dwNumFaces*6];    DWORD dwNumEdges = 0;    // For each face    for( DWORD i = 0; i < dwNumFaces; ++i )    {        WORD wFace0 = pIndices[3*i+0];        WORD wFace1 = pIndices[3*i+1];        WORD wFace2 = pIndices[3*i+2];	D3DXVECTOR3 v0 = pVertices[wFace0].p;	D3DXVECTOR3 v1 = pVertices[wFace1].p;	D3DXVECTOR3 v2 = pVertices[wFace2].p;                // Transform vertices or transform light?        D3DXVECTOR3 vCross1(v2-v1);        D3DXVECTOR3 vCross2(v1-v0);        D3DXVECTOR3 vNormal;        D3DXVec3Cross( &vNormal, &vCross1, &vCross2 );		D3DXVec3Normalize(&vNormal,&vNormal);        if( D3DXVec3Dot( &vNormal, &vLight ) >= 0.0f )        {            addEdge( pEdges, dwNumEdges, wFace0, wFace1 );            addEdge( pEdges, dwNumEdges, wFace1, wFace2 );            addEdge( pEdges, dwNumEdges, wFace2, wFace0 );        }    }    for( i = 0; i < dwNumEdges; ++i )    {        D3DXVECTOR3 v1 = pVertices[pEdges[2*i+0]].p;        D3DXVECTOR3 v2 = pVertices[pEdges[2*i+1]].p;        D3DXVECTOR3 v3 = v1 - vLight*20;        D3DXVECTOR3 v4 = v2 - vLight*20;				if (m_dwNumVertices<100000-7){         m_pVertices[m_dwNumVertices++] = v1;        m_pVertices[m_dwNumVertices++] = v2;        m_pVertices[m_dwNumVertices++] = v3;        m_pVertices[m_dwNumVertices++] = v2;        m_pVertices[m_dwNumVertices++] = v4;        m_pVertices[m_dwNumVertices++] = v3;		}    }    // Delete the temporary edge list    delete[] pEdges;    return S_OK;}//-----------------------------------------------------------------------------// Name: addEdge()// Desc: Adds an edge to a list of silohuette edges of a shadow volume.//-----------------------------------------------------------------------------void ShadowVolume::addEdge( WORD* pEdges, DWORD& dwNumEdges, WORD v0, WORD v1 ){    //// Remove interior edges (which appear in the list twice)    //for( DWORD i = 0; i < dwNumEdges; ++i )    //{    //    if( ( pEdges[2*i+0] == v0 && pEdges[2*i+1] == v1 ) ||    //        ( pEdges[2*i+0] == v1 && pEdges[2*i+1] == v0 ) )    //    {    //        if( dwNumEdges > 1 )    //        {    //            pEdges[2*i+0] = pEdges[2*(dwNumEdges-1)+0];    //            pEdges[2*i+1] = pEdges[2*(dwNumEdges-1)+1];    //        }    //        --dwNumEdges;    //        return;    //    }    //}    pEdges[2*dwNumEdges+0] = v0;    pEdges[2*dwNumEdges+1] = v1;    dwNumEdges++;}void ShadowVolume::renderShadowToStencilBuffer( LPDIRECT3DDEVICE9 g_pd3dDevice, D3DXMATRIX m_mat ){    // Disable z-buffer writes (note: z-testing still occurs), and enable the    // stencil-buffer	DWORD shade,cull,zwrite,stencil,alpha;	g_pd3dDevice->GetRenderState( D3DRS_SHADEMODE, &shade );    g_pd3dDevice->GetRenderState( D3DRS_CULLMODE,  &cull );    g_pd3dDevice->GetRenderState( D3DRS_ZWRITEENABLE,     &zwrite );    g_pd3dDevice->GetRenderState( D3DRS_STENCILENABLE,    &stencil );    g_pd3dDevice->GetRenderState( D3DRS_ALPHABLENDENABLE, &alpha );    g_pd3dDevice->SetRenderState( D3DRS_ZWRITEENABLE,  FALSE );    g_pd3dDevice->SetRenderState( D3DRS_STENCILENABLE, TRUE );    // Dont bother with interpolating color    g_pd3dDevice->SetRenderState( D3DRS_SHADEMODE, D3DSHADE_FLAT );    // Set up stencil compare fuction, reference value, and masks.    // Stencil test passes if ((ref & mask) cmpfn (stencil & mask)) is true.    // Note: since we set up the stencil-test to always pass, the STENCILFAIL    // renderstate is really not needed.    g_pd3dDevice->SetRenderState( D3DRS_STENCILFUNC,  D3DCMP_ALWAYS );    g_pd3dDevice->SetRenderState( D3DRS_STENCILZFAIL, D3DSTENCILOP_KEEP );    g_pd3dDevice->SetRenderState( D3DRS_STENCILFAIL,  D3DSTENCILOP_KEEP );    // If z-test passes, inc/decrement stencil buffer value    g_pd3dDevice->SetRenderState( D3DRS_STENCILREF,       0x1 );    g_pd3dDevice->SetRenderState( D3DRS_STENCILMASK,      0xffffffff );    g_pd3dDevice->SetRenderState( D3DRS_STENCILWRITEMASK, 0xffffffff );    g_pd3dDevice->SetRenderState( D3DRS_STENCILPASS,      D3DSTENCILOP_INCR );    // Make sure that no pixels get drawn to the frame buffer    g_pd3dDevice->SetRenderState( D3DRS_ALPHABLENDENABLE, TRUE );    g_pd3dDevice->SetRenderState( D3DRS_SRCBLEND,  D3DBLEND_ZERO );    g_pd3dDevice->SetRenderState( D3DRS_DESTBLEND, D3DBLEND_ONE );	if( g_bTwoSidedStencilsAvailable == true )    {        // With 2-sided stencil, we can avoid rendering twice:        g_pd3dDevice->SetRenderState( D3DRS_TWOSIDEDSTENCILMODE, TRUE );        g_pd3dDevice->SetRenderState( D3DRS_CCW_STENCILFUNC,  D3DCMP_ALWAYS );        g_pd3dDevice->SetRenderState( D3DRS_CCW_STENCILZFAIL, D3DSTENCILOP_KEEP );        g_pd3dDevice->SetRenderState( D3DRS_CCW_STENCILFAIL,  D3DSTENCILOP_KEEP );        g_pd3dDevice->SetRenderState( D3DRS_CCW_STENCILPASS, D3DSTENCILOP_DECR );        g_pd3dDevice->SetRenderState( D3DRS_CULLMODE,  D3DCULL_NONE );        // Draw both sides of shadow volume in stencil/z only        g_pd3dDevice->SetTransform( D3DTS_WORLD, &m_mat );        render( g_pd3dDevice );        g_pd3dDevice->SetRenderState( D3DRS_TWOSIDEDSTENCILMODE, FALSE );    }    else    {        // Draw front-side of shadow volume in stencil/z only        g_pd3dDevice->SetTransform( D3DTS_WORLD, &m_mat );        render( g_pd3dDevice );        // Now reverse cull order so back sides of shadow volume are written.        g_pd3dDevice->SetRenderState( D3DRS_CULLMODE, D3DCULL_CW );        // Decrement stencil buffer value        g_pd3dDevice->SetRenderState( D3DRS_STENCILPASS, D3DSTENCILOP_DECR );        // Draw back-side of shadow volume in stencil/z only        g_pd3dDevice->SetTransform( D3DTS_WORLD, &m_mat );        render( g_pd3dDevice );    }    // Restore render states	g_pd3dDevice->SetRenderState( D3DRS_SHADEMODE, shade );    g_pd3dDevice->SetRenderState( D3DRS_CULLMODE,  cull );    g_pd3dDevice->SetRenderState( D3DRS_ZWRITEENABLE,     zwrite );    g_pd3dDevice->SetRenderState( D3DRS_STENCILENABLE,    stencil );    g_pd3dDevice->SetRenderState( D3DRS_ALPHABLENDENABLE, alpha );}void ShadowVolume::renderShadowToScene( LPDIRECT3DDEVICE9 g_pd3dDevice ){     // Set render states    g_pd3dDevice->SetRenderState( D3DRS_ZENABLE,          FALSE );    g_pd3dDevice->SetRenderState( D3DRS_STENCILENABLE,    TRUE );    g_pd3dDevice->SetRenderState( D3DRS_ALPHABLENDENABLE, TRUE );    g_pd3dDevice->SetRenderState( D3DRS_SRCBLEND,  D3DBLEND_SRCALPHA );    g_pd3dDevice->SetRenderState( D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA );    g_pd3dDevice->SetTextureStageState( 0, D3DTSS_COLORARG1, D3DTA_TEXTURE );    g_pd3dDevice->SetTextureStageState( 0, D3DTSS_COLORARG2, D3DTA_DIFFUSE );    g_pd3dDevice->SetTextureStageState( 0, D3DTSS_COLOROP,   D3DTOP_MODULATE );    g_pd3dDevice->SetTextureStageState( 0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE );    g_pd3dDevice->SetTextureStageState( 0, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE );    g_pd3dDevice->SetTextureStageState( 0, D3DTSS_ALPHAOP,   D3DTOP_MODULATE );    // Only write where stencil value >= 1 (count indicates # of shadows that    // overlap that pixel)    g_pd3dDevice->SetRenderState( D3DRS_STENCILREF,  0x1 );    g_pd3dDevice->SetRenderState( D3DRS_STENCILFUNC, D3DCMP_LESSEQUAL );    g_pd3dDevice->SetRenderState( D3DRS_STENCILPASS, D3DSTENCILOP_KEEP );    // Draw a big, gray square	g_pd3dDevice->SetFVF( ShadowVertex::FVF_Flags );    g_pd3dDevice->SetStreamSource( 0, m_pBigSquareVB, 0, sizeof(ShadowVertex) );    g_pd3dDevice->DrawPrimitive( D3DPT_TRIANGLESTRIP, 0, 2 );    // Restore render states    g_pd3dDevice->SetRenderState( D3DRS_ZENABLE,          TRUE );    g_pd3dDevice->SetRenderState( D3DRS_STENCILENABLE,    FALSE );    g_pd3dDevice->SetRenderState( D3DRS_ALPHABLENDENABLE, FALSE );}
wusher, I am going through the plane, when I render it in wireframe I can see it go through my ground and down below that pretty far.

Jeff.
reverse the vertex winding of the extruded faces based on which face of the edge is back facing...

since edges are shared between triangles, that means each edges has 2 correct windings, one for each triangle.

i hope this is making some sense.
Uhm, not really sure. Are you saying when I add the vertices like this:

m_pVertices[m_dwNumVertices++] = v1;
m_pVertices[m_dwNumVertices++] = v2;
m_pVertices[m_dwNumVertices++] = v3;
m_pVertices[m_dwNumVertices++] = v2;
m_pVertices[m_dwNumVertices++] = v4;
m_pVertices[m_dwNumVertices++] = v3;

Its not always the case and that some should be reversed?? If so, how do I check that? Or am I totally not getting what your saying.

Thanks for helping, this is insane. Also, if I leave in the function to remove vertices that appear twice in the list you actually see the entire wireframe shadow volume and it looks all messed up. I leave that out and you get what you see in the screenshot. Just another note.

Thanks
Jeff.
okay,

there seems to be a lot wrong with how you're doing the whole
buildShadowVolume thing... you appear to be adding 3 edges for each triangle facing the light... it really should be closer to 1.

it should go something like:

build an edge list, each edge stores a pointer to 2 verts and 2 triangles.

for every edge
if egde->tri1 facing is not equal to edge->tri2 facing
add the 2 verts of this edge and the 2 verts of this edge
extruded to infinity.

Hi,

It looks very much like a problem concerning polygon list.

I noticed that on certain models my shadow volume had holes of that type.

The problem turned out to be that the shadow volume algorithm had to operate on a triangle list - the mesh I was using contained triangle fans and strips.

I think that you should try to use separate models for the shadow and the model. If you try to ensure that the model used for the shadow contains no triangle strips or fans then it should work.

This topic is closed to new replies.

Advertisement