I've been following the Shadows chapter of Math for 3D Game Programming, but for some reason I'm ending up with a weird problem that I can't seem to find the cause of. Here's a screenshot of what I'm seeing:

http://i.imgur.com/At3Xa.png

The black lines are the shadow volumes for all of the objects in my scene, using a single infinite light source. They're all coming from the stencil buffer (I've dumped the depth buffer between my ambient pass, my shadowing pass, and my lighting pass and it's never modified after the ambient), but I don't understand why I'm seeing the volume instead of just shadows where they intersect geometry. Any ideas what's going on? I'm more than happy to post whatever code would be relevant if that would help. Thanks in advance!

Here's another shot with a point light source: http://i.imgur.com/7A1Af.png

Hm, it looks like maybe decrementing the stencil buffer isn't working right. I'm doing this method:


// increment for front faces
glCullFace(GL_BACK);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
renderable.render_shadow(light, false);

// decrement for back faces
glCullFace(GL_FRONT);
glStencilOp(GL_KEEP, GL_KEEP, GL_DECR);
renderable.render_shadow(light, false);


If I comment out the back face step, it looks the same as it does with it kept in. However, if I comment out the front face step then the scene is completely unshadowed. I would expect that if the DECR was happening as it's supposed to, then I would still see the volume as the stencil buffer would be non-zero there.

Hi!
What are you doing with your depth buffer? The fact that commenting out the backface step doesn’t change anything sounds to me like the stencil test never passes. Perhaps the fragments were discarded by the depth test. Do you have turned off the writing of depth values in the front face pass? (glDepthMask(GL_FALSE);)

I do. When I make my shadow rendering pass, I turn off writes to both the color buffer and the depth buffer. I've read out the depth buffer to disk after the pass completes and it looks the same to me as it does before I start.

Alright, let’s see… What about the winding order of your triangles? Is what GL thinks to be a front face truly what’s in your model a front face? (glFrontFace(...))

I’d guess that something is wrong with your state setup. Can you tell us more about that or show us some lines of your code? Looking at code makes it easier to spot those little bugs. Maybe you can find your mistake by comparing your code to a working implementation. NeHe has a tutorial here.

Perhaps you’d like to read on zFail shadow volumes. They also work if your camera stands inside a shadow volume. The classical stencil shadows will fail then, since the front face of the volume can’t increment the stencil buffer. There is an implementation on paulsproject.

Yeah, I think it's probably going to be easiest if I stick some code in here. I'm using two FBOs, one for the ambient pass and one for the lighting pass, which I combine at the end in a separate shader. Both FBOs share the same depth/stencil buffer, which I've verified is D24S8, and both are checked for completeness. I've checked both color buffers separately and everything looks ok on that end, so I don't think the FBOs are related to the issue (though I can't discount them). My stencil is also cleared to 1 rather than 0 at the moment, but that should be taken into account when I do the lighting pass and check for a 1 rather than a 0.

This is my render function:


// render the ambient (filling the depth buffer)
glBindFramebuffer(GL_FRAMEBUFFER, _fbo[AmbientBuffer]);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
render_ambient(map);
glBindFramebuffer(GL_FRAMEBUFFER, 0);

// render the detail
glBindFramebuffer(GL_FRAMEBUFFER, _fbo[DetailBuffer]);
glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

// fill the stencil buffer with shadows
if(Light::lighting_enabled()) {
render_shadows(map);
}

// only render where the stencil is 0 and the depth is equal (only modify the color buffer)
glDepthMask(GL_FALSE);
glDepthFunc(GL_EQUAL);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glStencilFunc(GL_EQUAL, 1, ~0);
//glEnable(GL_BLEND);
//glBlendFunc(GL_ONE, GL_ONE);
render_detail(map);
//glDisable(GL_BLEND);
glStencilFunc(GL_ALWAYS, 0, ~0);
glDepthFunc(GL_LEQUAL);
glDepthMask(GL_TRUE);

glBindFramebuffer(GL_FRAMEBUFFER, 0);


The blending code is commented out because I'm doing that in the final stage shader.

This is the render_shadows method:


// disable color and depth writes
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);

// setup the stencil and depth functions
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glStencilFunc(GL_ALWAYS, 0, ~0);
glDepthFunc(GL_LESS);

glEnable(GL_CULL_FACE);

// offset the shadows
/*glEnable(GL_POLYGON_OFFSET_FILL);
glPolygonOffset(0.0f, 100.0f);*/

BOOST_FOREACH(boost::shared_ptr<Light> light, map.lights()) {
if(!light->enabled()) {
continue;
}

BOOST_FOREACH(boost::shared_ptr<Renderable> renderable, _light_renderables) {
if(renderable->has_shadow()) {
render_shadow(*renderable, *light);
}
}
}

glCullFace(GL_BACK);

//glDisable(GL_POLYGON_OFFSET_FILL);

glDepthFunc(GL_LEQUAL);
glStencilFunc(GL_ALWAYS, 0, ~0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

// re-enable color and depth writes
glDepthMask(GL_TRUE);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);


and this is the render_shadow method that does each object per-light:


glEnable(GL_CULL_FACE);

// increment for front faces
glCullFace(GL_BACK);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
renderable.render_shadow(shader, light, false);

// decrement for back faces
glCullFace(GL_FRONT);
glStencilOp(GL_KEEP, GL_KEEP, GL_DECR);
renderable.render_shadow(shader, light, false);


I'm not currently doing any of the volume capping, but I don't think that should be causing what I'm seeing.

This is the renderable's render_shadow method, where I determine the silhouette:


Matrix4 matrix;
transform(matrix);

Renderer::instance().push_model_matrix();
Renderer::instance().multiply_model_matrix(matrix);

// Mathematics for 3D Game Programming and Computer Graphics, section 10.3
size_t vstart = 0;
for(size_t i=0; i<_model->mesh_count(); ++i) {
const MD5Mesh& mesh(_model->mesh(i));

// determine the mesh silhouette
std::vector<MD5Edge> silhouette;
for(size_t j=0; j<mesh.edge_count(); ++j) {
const MD5Edge& edge(mesh.edge(j));

// calculate the plane equation of the faces using the face normal
Point4 f1, f2;
if(edge.t1 >= 0) {
const MD5Triangle& t(mesh.triangle(edge.t1));
const Vertex &v0(_vertices[vstart + t.v1]), &v1(_vertices[vstart + t.v2]), &v2(_vertices[vstart + t.v3]);
const Point3 q1(v1.position - v0.position), q2(v2.position - v0.position);
const Point3 n1(q1 ^ q2);
f1 = Point4(n1, -n1 * _vertices[vstart + t.v1].position);
}
if(edge.t2 >= 0) {
const MD5Triangle& t(mesh.triangle(edge.t2));
const Vertex &v0(_vertices[vstart + t.v1]), &v1(_vertices[vstart + t.v2]), &v2(_vertices[vstart + t.v3]);
const Point3 q1(v1.position - v0.position), q2(v2.position - v0.position);
const Point3 n2(q1 ^ q2);
f2 = Point4(n2, -n2 * _vertices[vstart + t.v2].position);
}

// sort out which of the edges, if any, are facing the light (with the light converted to object-space)
float dt1=0, dt2=0;
if(typeid(light) == typeid(DirectionalLight)) {
const DirectionalLight& directional(dynamic_cast<const DirectionalLight&>(light));
const Point4 direction(matrix * directional.direction());
dt1 = f1 * direction;
dt2 = f2 * direction;
} else if(typeid(light) == typeid(PositionalLight)) {
const PositionalLight& positional(dynamic_cast<const PositionalLight&>(light));
const Point4 position(matrix * positional.position4());
dt1 = f1 * position;
dt2 = f2 * position;
} else if(typeid(light) == typeid(SpotLight)) {
const SpotLight& spot(dynamic_cast<const SpotLight&>(light));
const Point4 position(matrix * spot.position4());
dt1 = f1 * position;
dt2 = f2 * position;
}

if(edge.t1 >= 0 && edge.t2 >= 0 && ((dt1 >= 0 && dt2 < 0) || (dt1 < 0 && dt2 >= 0))) {
// two-winged edges are silhouette edges if one triangle faces towards the light and the other way
silhouette.push_back(edge);
} else if((edge.t1 < 0 || edge.t2 < 0) && dt1 >= 0) {
// one-winged edges are only a silhouette edge if the *first* triangle faces the light
silhouette.push_back(edge);
}
}

render_silhouette(shader, mesh, silhouette, vstart, light, cap);
vstart += mesh.vertex_count();
}

Renderer::instance().pop_model_matrix();


And this is the rendering of the silhouette:


// Mathematics for 3D Game Programming and Computer Graphics, section 10.3/4

// build the vertex buffer
const size_t vcount = silhouette.size() * 3;
boost::shared_array<float> varray(new float[vcount * 3]), narray(new float[vcount * 3]);
float *v = varray.get(), *n = narray.get();
for(size_t i=0; i<silhouette.size(); ++i) {
const MD5Edge& edge(silhouette);
const Vertex &v1(_vertices[vstart + edge.v1]), &v2(_vertices[vstart + edge.v2]);

const size_t idx = i * 3 * 3;

// calculate the face normal for the first triangle
const MD5Triangle& t(mesh.triangle(edge.t1));
const Vertex &tv0(_vertices[vstart + t.v1]), &tv1(_vertices[vstart + t.v2]), &tv2(_vertices[vstart + t.v3]);
const Point3 q1(tv1.position - tv0.position), q2(tv2.position - tv0.position);
const Point3 normal(q1 ^ q2);

*(v + idx + 0) = v1.position.x;
*(v + idx + 1) = v1.position.y;
*(v + idx + 2) = v1.position.z;

*(v + idx + 3) = v2.position.x;
*(v + idx + 4) = v2.position.y;
*(v + idx + 5) = v2.position.z;

// 3rd vertex doesn't matter
*(v + idx + 6) = 0.0f;
*(v + idx + 7) = 0.0f;
*(v + idx + 8) = 0.0f;

// push the face normal to all vertices
*(n + idx + 0) = *(n + idx + 3) = *(n + idx + 6) = normal.x;
*(n + idx + 1) = *(n + idx + 4) = *(n + idx + 7) = normal.y;
*(n + idx + 2) = *(n + idx + 5) = *(n + idx + 8) = normal.z;
}

// setup the vertex array
glBindBuffer(GL_ARRAY_BUFFER, _shadow_vbo[ShadowVertexArray]);
glBufferData(GL_ARRAY_BUFFER, vcount * 3 * sizeof(float), varray.get(), GL_DYNAMIC_DRAW);

glBindBuffer(GL_ARRAY_BUFFER, _shadow_vbo[ShadowNormalArray]);
glBufferData(GL_ARRAY_BUFFER, vcount * 3 * sizeof(float), narray.get(), GL_DYNAMIC_DRAW);

shader.begin();
shader.uniform1i("cap", cap);
Renderer::instance().init_shader_matrices(shader);
Renderer::instance().init_shader_light(shader, light);

// get the attribute locations
GLint vloc = shader.attrib_location("vertex");
GLint nloc = shader.attrib_location("normal");

// render the silhouette
glEnableVertexAttribArray(vloc);
glEnableVertexAttribArray(nloc);
glBindBuffer(GL_ARRAY_BUFFER, _shadow_vbo[ShadowNormalArray]);
glVertexAttribPointer(nloc, 3, GL_FLOAT, GL_FALSE, 0, 0);

glBindBuffer(GL_ARRAY_BUFFER, _shadow_vbo[ShadowVertexArray]);
glVertexAttribPointer(vloc, 3, GL_FLOAT, GL_FALSE, 0, 0);

glDrawArrays(GL_TRIANGLES, 0, vcount);
glDisableVertexAttribArray(nloc);
glDisableVertexAttribArray(vloc);

shader.end();


I'm doing the actual edge extrusion in a geometry shader (this one is for infinite lights):


#version 150

uniform mat4 mvp, modelview;

// true if we need to cap the shadow
uniform bool cap;

// these are in eye-space
uniform vec4 light_position;

layout(triangles) in;
layout(triangle_strip, max_vertices=3) out;

in vec3 geom_normal[3];

out vec4 frag_color;

void main()
{
// Mathematics for 3D Game Programming and Computer Graphics, section 10.4

// convert the light to object-space
vec4 L = inverse(modelview) * light_position;

// extrude the silhouette
vec4 vertices[3];
float dt = dot(geom_normal[0], L.xyz);
if(dt > 0) {
frag_color = vec4(1.0, 0.0, 0.0, 1.0);
vertices[0] = vec4(gl_in[1].gl_Position.xyz, 1.0);
vertices[1] = vec4(gl_in[0].gl_Position.xyz, 1.0);
vertices[2] = vec4(0.0);
} else {
frag_color = vec4(0.0, 0.0, 1.0, 1.0);
vertices[0] = vec4(gl_in[0].gl_Position.xyz, 1.0);
vertices[1] = vec4(gl_in[1].gl_Position.xyz, 1.0);
vertices[2] = vec4(0.0);
}

for(int i=0; i<vertices.length(); ++i) {
gl_Position = mvp * (vertices.w * (vertices + L) - L);
EmitVertex();
}
EndPrimitive();
}


I have a fragment shader attached to that geometry shader so I can render the shadow volume separately and it looks to me like the normals are correct when I color the extruded geometry with them and changing the face culling when I render it looks like what I would expect. Front faces where you want front faces and so on.

I really appreciate the help with this. Every example I've found for how to do this seems to be "do it this way and it just works" with no hints on how to debug things when they don't end up working.

Hm, I’ve seen a few small things, but they are likely just there because the code is still under construction.

• If you don’t need the stencil test, better turn it off: glDisable(GL_STENCIL_TEST)
  
• You have some redundant states (e.g. before the foreach loop)
  

Things you can try to further narrow down the problem:

• Instead of the shadow volumes render a sphere that intersects with a wall and let its front face increment the stencil values and the back face decrement. This should give you a shadow where the sphere intersects the wall. This way you can see, whether your state setup is causing the problem or the rendering code of your shadow volumes.
  
• If this doesn’t work use glReadPixel to read the stencil buffer to the CPU, create a color array that maps the stencil values to colors and render this with glDrawPixels. (In D3D is a way to read directly from the stencil buffer in a shader. I’d assume that GL can do this too.)
  
• Is it possible that shadow volume faces are drawn twice? (You could see this, if you have a look at the values in the stencil buffer.)
  
• Don’t use FBOs for now. Try to get it running with the backbuffer, then all in a single FBO and then do the compositing with multiple FBOs (step by step).
  
• The render_detail pass should fill the depth buffer anew. GL_EQUAL should be avoided if possible, since it can be very evil due to precision problems. So, clear it and set glDepthFunc(GL_LEQUAL) and glDepthMask(GL_TRUE). If you got it working you can try with GL_EQUAL again.
  

Let us know what happens if you do those experiments. Good luck!

Ok, I think I've got some meaningful results here.

• Instead of the shadow volumes render a sphere that intersects with a wall and let its front face increment the stencil values and the back face decrement. This should give you a shadow where the sphere intersects the wall. This way you can see, whether your state setup is causing the problem or the rendering code of your shadow volumes.
  

I rendered a cube bisecting the back wall instead of the shadow volumes and there was no shadow at all. I tried again using the stencil_two_side extension and got the same result, no shadow. Very curious.

• If this doesn’t work use glReadPixel to read the stencil buffer to the CPU, create a color array that maps the stencil values to colors and render this with glDrawPixels. (In D3D is a way to read directly from the stencil buffer in a shader. I’d assume that GL can do this too.)
  
• Is it possible that shadow volume faces are drawn twice? (You could see this, if you have a look at the values in the stencil buffer.)
  

I've used this code to render out the stencil buffer as my detail shader texture:


boost::shared_array<unsigned char> stencil(new unsigned char[window_width() * window_height() * 4]);
glReadPixels(0, 0, _window->w, _window->h, GL_DEPTH_STENCIL, GL_UNSIGNED_INT_24_8, stencil.get());
glBindTexture(GL_TEXTURE_2D, _tbo[DetailBuffer]);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, window_width(), window_height(), 0, GL_RGBA, GL_UNSIGNED_BYTE, stencil.get());


Having the shader map the red component to all 3 colors (gba seem to all be from the depth buffer) I got this: http://i.imgur.com/95ZXM.png

I tried the same thing rendering the cube and there were no white pixels as I moved toward the back wall.

• Don’t use FBOs for now. Try to get it running with the backbuffer, then all in a single FBO and then do the compositing with multiple FBOs (step by step).
  

I sent everything to framebuffer 0 and re-enabled blending, but got the same result as before.

• The render_detail pass should fill the depth buffer anew. GL_EQUAL should be avoided if possible, since it can be very evil due to precision problems. So, clear it and set glDepthFunc(GL_LEQUAL) and glDepthMask(GL_TRUE). If you got it working you can try with GL_EQUAL again.
  

Unfortunately, this didn't change anything either. I figured that what you're saying would be a better idea than turning the mask off and testing for equality, but at this point I'm trying to stick as closely to what I had read in case leaving something out turns out to be a source of the problem. Though it seems like maybe the setup is where I've gone wrong, given that the cube didn't seem to affect the stencil values. Could it be that I am actually doubling up on edges and causing the volume geometry to be rendered twice? They're not affecting the depth buffer, so it would have to be an odd number of renderings in order to cut out the volume like that, wouldn't it?

Just did a pass through my silhouette edges to see if any 2 vertices show up twice in one edge, regardless of ordering, and it doesn't seem to occur, so I don't think that I'm rendering the volume geometry more than once. I could still be wrong about that, but it would have to be happening somewhere else.