Jump to content

  • Log In with Google      Sign In   
  • Create Account

What's wrong with my stencil shadows?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
16 replies to this topic

#1 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 07 January 2012 - 10:59 AM

I've been following the Shadows chapter of Math for 3D Game Programming, but for some reason I'm ending up with a weird problem that I can't seem to find the cause of. Here's a screenshot of what I'm seeing:

http://i.imgur.com/At3Xa.png

The black lines are the shadow volumes for all of the objects in my scene, using a single infinite light source. They're all coming from the stencil buffer (I've dumped the depth buffer between my ambient pass, my shadowing pass, and my lighting pass and it's never modified after the ambient), but I don't understand why I'm seeing the volume instead of just shadows where they intersect geometry. Any ideas what's going on? I'm more than happy to post whatever code would be relevant if that would help. Thanks in advance!

Sponsor:

#2 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 07 January 2012 - 11:28 AM

Here's another shot with a point light source: http://i.imgur.com/7A1Af.png

#3 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 07 January 2012 - 06:17 PM

Hm, it looks like maybe decrementing the stencil buffer isn't working right. I'm doing this method:

		    // increment for front faces
		    glCullFace(GL_BACK);
		    glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
		    renderable.render_shadow(light, false);

		    // decrement for back faces
		    glCullFace(GL_FRONT);
		    glStencilOp(GL_KEEP, GL_KEEP, GL_DECR);
		    renderable.render_shadow(light, false);

If I comment out the back face step, it looks the same as it does with it kept in. However, if I comment out the front face step then the scene is completely unshadowed. I would expect that if the DECR was happening as it's supposed to, then I would still see the volume as the stencil buffer would be non-zero there.

#4 Tsus   Members   -  Reputation: 1036

Like
0Likes
Like

Posted 08 January 2012 - 11:18 AM

Hi!
What are you doing with your depth buffer? The fact that commenting out the backface step doesn’t change anything sounds to me like the stencil test never passes. Perhaps the fragments were discarded by the depth test. Do you have turned off the writing of depth values in the front face pass? (glDepthMask(GL_FALSE);)

#5 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 08 January 2012 - 02:14 PM

I do. When I make my shadow rendering pass, I turn off writes to both the color buffer and the depth buffer. I've read out the depth buffer to disk after the pass completes and it looks the same to me as it does before I start.

#6 Tsus   Members   -  Reputation: 1036

Like
0Likes
Like

Posted 08 January 2012 - 03:56 PM

Alright, let’s see… What about the winding order of your triangles? Is what GL thinks to be a front face truly what’s in your model a front face? (glFrontFace(...))

I’d guess that something is wrong with your state setup. Can you tell us more about that or show us some lines of your code? Looking at code makes it easier to spot those little bugs. Posted Image Maybe you can find your mistake by comparing your code to a working implementation. NeHe has a tutorial here.

Perhaps you’d like to read on zFail shadow volumes. They also work if your camera stands inside a shadow volume. The classical stencil shadows will fail then, since the front face of the volume can’t increment the stencil buffer. There is an implementation on paulsproject.

#7 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 08 January 2012 - 04:17 PM

Yeah, I think it's probably going to be easiest if I stick some code in here. I'm using two FBOs, one for the ambient pass and one for the lighting pass, which I combine at the end in a separate shader. Both FBOs share the same depth/stencil buffer, which I've verified is D24S8, and both are checked for completeness. I've checked both color buffers separately and everything looks ok on that end, so I don't think the FBOs are related to the issue (though I can't discount them). My stencil is also cleared to 1 rather than 0 at the moment, but that should be taken into account when I do the lighting pass and check for a 1 rather than a 0.

This is my render function:

    // render the ambient (filling the depth buffer)
    glBindFramebuffer(GL_FRAMEBUFFER, _fbo[AmbientBuffer]);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
    render_ambient(map);
    glBindFramebuffer(GL_FRAMEBUFFER, 0);

    // render the detail
    glBindFramebuffer(GL_FRAMEBUFFER, _fbo[DetailBuffer]);
    glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

    // fill the stencil buffer with shadows
    if(Light::lighting_enabled()) {
	   render_shadows(map);
    }

    // only render where the stencil is 0 and the depth is equal (only modify the color buffer)
    glDepthMask(GL_FALSE);
    glDepthFunc(GL_EQUAL);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
    glStencilFunc(GL_EQUAL, 1, ~0);
    //glEnable(GL_BLEND);
    //glBlendFunc(GL_ONE, GL_ONE);
	    render_detail(map);
    //glDisable(GL_BLEND);
    glStencilFunc(GL_ALWAYS, 0, ~0);
    glDepthFunc(GL_LEQUAL);
    glDepthMask(GL_TRUE);

    glBindFramebuffer(GL_FRAMEBUFFER, 0);

The blending code is commented out because I'm doing that in the final stage shader.

This is the render_shadows method:

    // disable color and depth writes
    glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
    glDepthMask(GL_FALSE);

    // setup the stencil and depth functions
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
    glStencilFunc(GL_ALWAYS, 0, ~0);
    glDepthFunc(GL_LESS);

    glEnable(GL_CULL_FACE);

    // offset the shadows
    /*glEnable(GL_POLYGON_OFFSET_FILL);
    glPolygonOffset(0.0f, 100.0f);*/

    BOOST_FOREACH(boost::shared_ptr<Light> light, map.lights()) {
	    if(!light->enabled()) {
		    continue;
	    }

	    BOOST_FOREACH(boost::shared_ptr<Renderable> renderable, _light_renderables) {
		    if(renderable->has_shadow()) {
			    render_shadow(*renderable, *light);
		    }
	    }
    }

    glCullFace(GL_BACK);

    //glDisable(GL_POLYGON_OFFSET_FILL);

    glDepthFunc(GL_LEQUAL);
    glStencilFunc(GL_ALWAYS, 0, ~0);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

    // re-enable color and depth writes
    glDepthMask(GL_TRUE);
    glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

and this is the render_shadow method that does each object per-light:

		    glEnable(GL_CULL_FACE);

		    // increment for front faces
		    glCullFace(GL_BACK);
		    glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
		    renderable.render_shadow(shader, light, false);

		    // decrement for back faces
		    glCullFace(GL_FRONT);
		    glStencilOp(GL_KEEP, GL_KEEP, GL_DECR);
		    renderable.render_shadow(shader, light, false);

I'm not currently doing any of the volume capping, but I don't think that should be causing what I'm seeing.

This is the renderable's render_shadow method, where I determine the silhouette:

    Matrix4 matrix;
    transform(matrix);

    Renderer::instance().push_model_matrix();
    Renderer::instance().multiply_model_matrix(matrix);

    // Mathematics for 3D Game Programming and Computer Graphics, section 10.3
    size_t vstart = 0;
    for(size_t i=0; i<_model->mesh_count(); ++i) {
	    const MD5Mesh& mesh(_model->mesh(i));

	    // determine the mesh silhouette
	    std::vector<MD5Edge> silhouette;
	    for(size_t j=0; j<mesh.edge_count(); ++j) {
		    const MD5Edge& edge(mesh.edge(j));

		    // calculate the plane equation of the faces using the face normal
		    Point4 f1, f2;
		    if(edge.t1 >= 0) {
			    const MD5Triangle& t(mesh.triangle(edge.t1));
			    const Vertex &v0(_vertices[vstart + t.v1]), &v1(_vertices[vstart + t.v2]), &v2(_vertices[vstart + t.v3]);
			    const Point3 q1(v1.position - v0.position), q2(v2.position - v0.position);
			    const Point3 n1(q1 ^ q2);
			    f1 = Point4(n1, -n1 * _vertices[vstart + t.v1].position);
		    }
		    if(edge.t2 >= 0) {
			    const MD5Triangle& t(mesh.triangle(edge.t2));
			    const Vertex &v0(_vertices[vstart + t.v1]), &v1(_vertices[vstart + t.v2]), &v2(_vertices[vstart + t.v3]);
			    const Point3 q1(v1.position - v0.position), q2(v2.position - v0.position);
			    const Point3 n2(q1 ^ q2);
			    f2 = Point4(n2, -n2 * _vertices[vstart + t.v2].position);
		    }

		    // sort out which of the edges, if any, are facing the light (with the light converted to object-space)
		    float dt1=0, dt2=0;
		    if(typeid(light) == typeid(DirectionalLight)) {
			    const DirectionalLight& directional(dynamic_cast<const DirectionalLight&>(light));
			    const Point4 direction(matrix * directional.direction());
			    dt1 = f1 * direction;
			    dt2 = f2 * direction;
		    } else if(typeid(light) == typeid(PositionalLight)) {
			    const PositionalLight& positional(dynamic_cast<const PositionalLight&>(light));
			    const Point4 position(matrix * positional.position4());
			    dt1 = f1 * position;
			    dt2 = f2 * position;
		    } else if(typeid(light) == typeid(SpotLight)) {
			    const SpotLight& spot(dynamic_cast<const SpotLight&>(light));
			    const Point4 position(matrix * spot.position4());
			    dt1 = f1 * position;
			    dt2 = f2 * position;
		    }

		    if(edge.t1 >= 0 && edge.t2 >= 0 && ((dt1 >= 0 && dt2 < 0) || (dt1 < 0 && dt2 >= 0))) {
			    // two-winged edges are silhouette edges if one triangle faces towards the light and the other way
			    silhouette.push_back(edge);
		    } else if((edge.t1 < 0 || edge.t2 < 0) && dt1 >= 0) {
			    // one-winged edges are only a silhouette edge if the *first* triangle faces the light
			    silhouette.push_back(edge);
		    }
	    }

	    render_silhouette(shader, mesh, silhouette, vstart, light, cap);
	    vstart += mesh.vertex_count();
    }

    Renderer::instance().pop_model_matrix();

And this is the rendering of the silhouette:

    // Mathematics for 3D Game Programming and Computer Graphics, section 10.3/4

    // build the vertex buffer
    const size_t vcount = silhouette.size() * 3;
    boost::shared_array<float> varray(new float[vcount * 3]), narray(new float[vcount * 3]);
    float *v = varray.get(), *n = narray.get();
    for(size_t i=0; i<silhouette.size(); ++i) {
	    const MD5Edge& edge(silhouette[i]);
	    const Vertex &v1(_vertices[vstart + edge.v1]), &v2(_vertices[vstart + edge.v2]);

	    const size_t idx = i * 3 * 3;

	    // calculate the face normal for the first triangle
	    const MD5Triangle& t(mesh.triangle(edge.t1));
	    const Vertex &tv0(_vertices[vstart + t.v1]), &tv1(_vertices[vstart + t.v2]), &tv2(_vertices[vstart + t.v3]);
	    const Point3 q1(tv1.position - tv0.position), q2(tv2.position - tv0.position);
	    const Point3 normal(q1 ^ q2);

	    *(v + idx + 0) = v1.position.x;
	    *(v + idx + 1) = v1.position.y;
	    *(v + idx + 2) = v1.position.z;

	    *(v + idx + 3) = v2.position.x;
	    *(v + idx + 4) = v2.position.y;
	    *(v + idx + 5) = v2.position.z;

	    // 3rd vertex doesn't matter
	    *(v + idx + 6) = 0.0f;
	    *(v + idx + 7) = 0.0f;
	    *(v + idx + 8) = 0.0f;

	    // push the face normal to all vertices
	    *(n + idx + 0) = *(n + idx + 3) = *(n + idx + 6) = normal.x;
	    *(n + idx + 1) = *(n + idx + 4) = *(n + idx + 7) = normal.y;
	    *(n + idx + 2) = *(n + idx + 5) = *(n + idx + 8) = normal.z;
    }

    // setup the vertex array
    glBindBuffer(GL_ARRAY_BUFFER, _shadow_vbo[ShadowVertexArray]);
    glBufferData(GL_ARRAY_BUFFER, vcount * 3 * sizeof(float), varray.get(), GL_DYNAMIC_DRAW);

    glBindBuffer(GL_ARRAY_BUFFER, _shadow_vbo[ShadowNormalArray]);
    glBufferData(GL_ARRAY_BUFFER, vcount * 3 * sizeof(float), narray.get(), GL_DYNAMIC_DRAW);

    shader.begin();
    shader.uniform1i("cap", cap);
    Renderer::instance().init_shader_matrices(shader);
    Renderer::instance().init_shader_light(shader, light);

    // get the attribute locations
    GLint vloc = shader.attrib_location("vertex");
    GLint nloc = shader.attrib_location("normal");

    // render the silhouette
    glEnableVertexAttribArray(vloc);
    glEnableVertexAttribArray(nloc);
	    glBindBuffer(GL_ARRAY_BUFFER, _shadow_vbo[ShadowNormalArray]);
	    glVertexAttribPointer(nloc, 3, GL_FLOAT, GL_FALSE, 0, 0);

	    glBindBuffer(GL_ARRAY_BUFFER, _shadow_vbo[ShadowVertexArray]);
	    glVertexAttribPointer(vloc, 3, GL_FLOAT, GL_FALSE, 0, 0);

	    glDrawArrays(GL_TRIANGLES, 0, vcount);
    glDisableVertexAttribArray(nloc);
    glDisableVertexAttribArray(vloc);

    shader.end();

I'm doing the actual edge extrusion in a geometry shader (this one is for infinite lights):

#version 150

uniform mat4 mvp, modelview;

// true if we need to cap the shadow
uniform bool cap;

// these are in eye-space
uniform vec4 light_position;

layout(triangles) in;
layout(triangle_strip, max_vertices=3) out;

in vec3 geom_normal[3];

out vec4 frag_color;

void main()
{
    // Mathematics for 3D Game Programming and Computer Graphics, section 10.4

    // convert the light to object-space
    vec4 L = inverse(modelview) * light_position;

    // extrude the silhouette
    vec4 vertices[3];
    float dt = dot(geom_normal[0], L.xyz);
    if(dt > 0) {
	    frag_color = vec4(1.0, 0.0, 0.0, 1.0);
	    vertices[0] = vec4(gl_in[1].gl_Position.xyz, 1.0);
	    vertices[1] = vec4(gl_in[0].gl_Position.xyz, 1.0);
	    vertices[2] = vec4(0.0);
    } else {
	    frag_color = vec4(0.0, 0.0, 1.0, 1.0);
	    vertices[0] = vec4(gl_in[0].gl_Position.xyz, 1.0);
	    vertices[1] = vec4(gl_in[1].gl_Position.xyz, 1.0);
	    vertices[2] = vec4(0.0);
    }

    for(int i=0; i<vertices.length(); ++i) {
	    gl_Position = mvp * (vertices[i].w * (vertices[i] + L) - L);
	    EmitVertex();
    }
    EndPrimitive();
}

I have a fragment shader attached to that geometry shader so I can render the shadow volume separately and it looks to me like the normals are correct when I color the extruded geometry with them and changing the face culling when I render it looks like what I would expect. Front faces where you want front faces and so on.

I really appreciate the help with this. Every example I've found for how to do this seems to be "do it this way and it just works" with no hints on how to debug things when they don't end up working.

#8 Tsus   Members   -  Reputation: 1036

Like
0Likes
Like

Posted 08 January 2012 - 07:31 PM

Hm, I’ve seen a few small things, but they are likely just there because the code is still under construction.
  • If you don’t need the stencil test, better turn it off: glDisable(GL_STENCIL_TEST)
  • You have some redundant states (e.g. before the foreach loop)
Things you can try to further narrow down the problem:
  • Instead of the shadow volumes render a sphere that intersects with a wall and let its front face increment the stencil values and the back face decrement. This should give you a shadow where the sphere intersects the wall. This way you can see, whether your state setup is causing the problem or the rendering code of your shadow volumes.
  • If this doesn’t work use glReadPixel to read the stencil buffer to the CPU, create a color array that maps the stencil values to colors and render this with glDrawPixels. (In D3D is a way to read directly from the stencil buffer in a shader. I’d assume that GL can do this too.)
  • Is it possible that shadow volume faces are drawn twice? (You could see this, if you have a look at the values in the stencil buffer.)
  • Don’t use FBOs for now. Try to get it running with the backbuffer, then all in a single FBO and then do the compositing with multiple FBOs (step by step).
  • The render_detail pass should fill the depth buffer anew. GL_EQUAL should be avoided if possible, since it can be very evil due to precision problems. So, clear it and set glDepthFunc(GL_LEQUAL) and glDepthMask(GL_TRUE). If you got it working you can try with GL_EQUAL again.
Let us know what happens if you do those experiments. Good luck!

#9 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 08 January 2012 - 08:52 PM

Ok, I think I've got some meaningful results here.

  • Instead of the shadow volumes render a sphere that intersects with a wall and let its front face increment the stencil values and the back face decrement. This should give you a shadow where the sphere intersects the wall. This way you can see, whether your state setup is causing the problem or the rendering code of your shadow volumes.


I rendered a cube bisecting the back wall instead of the shadow volumes and there was no shadow at all. I tried again using the stencil_two_side extension and got the same result, no shadow. Very curious.

  • If this doesn’t work use glReadPixel to read the stencil buffer to the CPU, create a color array that maps the stencil values to colors and render this with glDrawPixels. (In D3D is a way to read directly from the stencil buffer in a shader. I’d assume that GL can do this too.)
  • Is it possible that shadow volume faces are drawn twice? (You could see this, if you have a look at the values in the stencil buffer.)


I've used this code to render out the stencil buffer as my detail shader texture:

boost::shared_array<unsigned char> stencil(new unsigned char[window_width() * window_height() * 4]);
glReadPixels(0, 0, _window->w, _window->h, GL_DEPTH_STENCIL, GL_UNSIGNED_INT_24_8, stencil.get());
glBindTexture(GL_TEXTURE_2D, _tbo[DetailBuffer]);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, window_width(), window_height(), 0, GL_RGBA, GL_UNSIGNED_BYTE, stencil.get());

Having the shader map the red component to all 3 colors (gba seem to all be from the depth buffer) I got this: http://i.imgur.com/95ZXM.png

I tried the same thing rendering the cube and there were no white pixels as I moved toward the back wall.

  • Don’t use FBOs for now. Try to get it running with the backbuffer, then all in a single FBO and then do the compositing with multiple FBOs (step by step).


I sent everything to framebuffer 0 and re-enabled blending, but got the same result as before.

  • The render_detail pass should fill the depth buffer anew. GL_EQUAL should be avoided if possible, since it can be very evil due to precision problems. So, clear it and set glDepthFunc(GL_LEQUAL) and glDepthMask(GL_TRUE). If you got it working you can try with GL_EQUAL again.


Unfortunately, this didn't change anything either. I figured that what you're saying would be a better idea than turning the mask off and testing for equality, but at this point I'm trying to stick as closely to what I had read in case leaving something out turns out to be a source of the problem. Though it seems like maybe the setup is where I've gone wrong, given that the cube didn't seem to affect the stencil values. Could it be that I am actually doubling up on edges and causing the volume geometry to be rendered twice? They're not affecting the depth buffer, so it would have to be an odd number of renderings in order to cut out the volume like that, wouldn't it?

#10 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 08 January 2012 - 09:04 PM

Just did a pass through my silhouette edges to see if any 2 vertices show up twice in one edge, regardless of ordering, and it doesn't seem to occur, so I don't think that I'm rendering the volume geometry more than once. I could still be wrong about that, but it would have to be happening somewhere else.

#11 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 08 January 2012 - 09:17 PM

Sorry, I did that cube test wrong. Doing it correctly I have the same effect as the shadow volumes. The entire cube volume is cute out of the scene rather than just shadowing on the wall.

#12 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 08 January 2012 - 09:37 PM

Sorry, removing some posts that came about as a result of me making some mistakes in testing. Both back and front face culling seem to be working.

#13 Tsus   Members   -  Reputation: 1036

Like
0Likes
Like

Posted 09 January 2012 - 03:16 AM

Hi again,

I think the results are helpful. Let us stick with the cube for now. It is the easier test case, where we have a better idea how the stencil buffer should look like.

Let’s find out whether both the front and back face pass work independently from each other correct. It would be interesting to see the debug output of your stencil buffer if you only render the front faces and let them increase the stencil value. After that, could you please take a second picture where you only render the back faces and let them increase the stencil value, too? It should look exactly the same except for the actual shadowed area. For a cube not intersecting a wall both should look exactly the same. Do they?

And we should only see the stencil values 1 and 2 (you still clear with 1, right?). Could you verify that as well? Perhaps, just iterate over your stencil array and see what you find in there.

#14 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 09 January 2012 - 02:50 PM

Unfortunately, I'm not in a place to test with the cube right now, but I did manage to dump the stencil buffer for my shadow volume (clearing with and checking against 0 rather than 1 here) to a text file with this:

const size_t s = window_width() * window_height();
boost::shared_array<unsigned char> stencil(new unsigned char[s]);
glPixelStorei(GL_PACK_ALIGNMENT, 1);
glReadPixels(0, 0, window_width(), window_height(), GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, stencil.get());
glPixelStorei(GL_PACK_ALIGNMENT, 4);

std::ofstream f("stencil.txt");
for(size_t y=0; y<window_height(); ++y) {
	for(size_t x=0; x<window_width(); ++x) {
		f << static_cast<int>(stencil[(y * window_width()) + x]) << " ";
	}
	f << std::endl;
}
f << std::endl;
f.close();

and I'm seeing values larger than 1. In fact I see all the way up to 9. It really is starting to seem like something is wrong with my decrementing.

#15 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 09 January 2012 - 03:53 PM

Just going to note that when I get home and can get back to the box shadows, I'm going to try with a clear value of 128 and check the variance of values. I'm guessing stencil indices can't be negative? It's possible that may be getting in the way here.

#16 Tsus   Members   -  Reputation: 1036

Like
0Likes
Like

Posted 09 January 2012 - 04:05 PM

Stencil values are clamped to 0 and 255 unless you tell them to wrap.
But I think that shouldn't be a problem, since you render the front faces first. You should only get into trouble if your objects aren't closed.

#17 Chozo   Members   -  Reputation: 176

Like
0Likes
Like

Posted 09 January 2012 - 06:34 PM

Ok, going back to the cube now that I'm home, still using 128 as a starting value, I see the full cube geometry with a 129 in the stencil where the cube is at. A little more messing around and it seems to be related to front face culling. For example, I can remove the increment and change the decrement to work on front faces and it decrements fine. I can also change the increment to work on back faces and it never increments. But I can render the cube with whichever face culling outside of shadows and it works fine.




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS