Per-polygon self shadowing problem with stencil shadows
I assume you combine your stencil shadows with drawing lighting with vertex-normals.
The problem occurs when normals are interpolated for smooth shading. If you draw your sphere with flat shading, as in that every triangle has only one normal, and not interpolated vertex normals, then it would look as expected. The reason for this is that stencil-volume edges are extruded between two triangles where one faces the light and one faces away from the light. With interpolated vertex-normals the normals used in the lighting calculation don't match the normals used to calculate the stencil shadow.
If every vertex that touches a triangle that is back-facing with respect to the light receives zero lighting, then the problem goes away.
You can fix it by, when you extrude the edges between back and front-facing polygons, instead of using the triangle plane normal to determine facing, looping over the three vertex normals in the triangle and checking if any of the vertex normals face the light. If at least one normal faces the light, then consider the triangle front-facing.
This will however cause your shadow volumes to shrink a bit, which may or may not be acceptable.
Another variation is to change the vertex-normals so that any vertex that touches a back-facing triangle receives zero lighting, which would keep your shadow volumes the same, but cause the shadow edge to grow a bit instead, as the shadow would be blurred into the lighted part instead of the light being blurred into the shadowed part.
Hi mv348, I have a very similar opinion to yours regarding regarding the self-shadowing back faces - it's there, and it's a problem that needs to be dealt with.
Fortunately, after some research I got to know the "Reverse-Extruded Shadow Volumes" technique by Renaldas Zioma.
In simple terms, to create the volume, instead of extruding the back faces and leaving the front faces in place, you do the opposite.
This technique is useful for two reasons: It works flawlessly with using a lower-detail mesh to generate your volume (so it's more performant); It also places any self-shadowing back faces exclusively in the ambient-lit region of the object - effectively hiding these hard edges if you use ambient shadows instead of the common "transparent stenciled-quad" shadows.
One way to do ambient shadows is to render your scene with lights and then do another pass (using the stencil) with ambient-only on all shadowed areas.
In any case, you can find literature for this technique here:
- http://aras-p.info/texts/revext.html (includes a demo)
- http://tog.acm.org/resources/shaderx/ (Shader X²: Tips & Tricks, Section V - Shadows)
With this solved, you're still left with having to avoid the Z-Fail method patented by Creative Labs. Have you had any progress on this?
Thanks for the detailed responses, Erik Rufelt, and Kryzon. I don't have time to go over your suggestions in detail at this moment, but will do so tomorrow.
With this solved, you're still left with having to avoid the Z-Fail method patented by Creative Labs. Have you had any progress on this?
I'm not sure what you are asking here. Is the depth-fail method for testing shadow-volumes considered a proprietary method of creative labs? O.o that would suck.
I'm surprised you weren't aware of this.
I can understand it, however. Patenting a graphic technique is perhaps foul play? (Imagine if normal-mapping and others were under the same condition).
Method for Rendering Shadows using a Shadow Volume and a Stencil Buffer
Stencil-shadows pretty much died because of this and from the fact that they don't scale well with higher-detail meshes, which was the natural prospect for game engines.
But with mobile specs and some lower-end consoles (the Wii, for instance) they can present a reasonable method again.
There are ways around camera intersection of shadow volumes that don't rely on depth-fail to render stencil shadows, but they're more complex and in some cases less accurate.
I'm not a lawyer. I was hoping something as simple as changing the stencil operators would not infringe the patent; After all, it wouldn't follow the description of the patent anymore.
Can anyone shed some light (no pun intended) on this? I'm referring to this post here (Shadow Volume SceneProcessor) where the OP claims at the end of his post of not infringing the patent by using different operators for the stencil function.
@mv348, do you use a high-poly mesh for rendering, but a low-poly mesh for shadows? Or do you use the same mesh for both?
As long as you're only extruding truly back-facing edges, this artefact shouldn't occur, or more specifically, it should only occur in areas that are un-lit, so you can't notice.
I'm not a lawyer. I was hoping something as simple as changing the stencil operators would not infringe the patent; After all, it wouldn't follow the description of the patent anymore.
Can anyone shed some light (no pun intended) on this? I'm referring to this post here (Shadow Volume SceneProcessor) where the OP claims at the end of his post of not infringing the patent by using different operators for the stencil function.
I am not a lawyer either, but I'd personally go ahead using modified versions of Z-fail stencil shadows like that one, without worrying too much.
If you read the wording on the creative labs patent, it's very specific in it's claims and therefore easy to circumvent.
For one, it explicitly claims rendering the front and back faces separately in two distinct passes. These days GPUs support two-sided stencil testing, where we can render the front and back faces in a single pass (with different stencil tests for each). This bypasses creative's claims and implements pretty much the same algorithm without using the method that's owned by them.
Thanks for confirming. This should make it much simpler then.
On the subject of lower-density meshes for generating the volumes, this technique was used on Shadow of the Colossus (PS2). They also blurred the resulting shadows to get soft edges. Their paper has some illustrations of it: Graphics used in Shadow of the Colossus
Thanks again to you all, for your helpful feedback and suggestions.
@mv348, do you use a high-poly mesh for rendering, but a low-poly mesh for shadows? Or do you use the same mesh for both?
Using the same for both. Its relatively low poly.
So I tried implementing what Erik Rufelt suggested:
You can fix it by, when you extrude the edges between back and front-facing polygons, instead of using the triangle plane normal to determine facing, looping over the three vertex normals in the triangle and checking if any of the vertex normals face the light. If at least one normal faces the light, then consider the triangle front-facing.
This will however cause your shadow volumes to shrink a bit, which may or may not be acceptable.
Pretty much did exactly what you said. Definitely fixes the self shadowing issue. However, the shadow shrinking produces a shadow with a jagged, per-polygon edge. So ironically it really just moves the same jagged artifacts off the self-shadows and puts them on the regular shadows. :\
@Hodgeman, you said:
As long as you're only extruding truly back-facing edges, this artefact shouldn't occur, or more specifically, it should only occur in areas that are un-lit, so you can't notice.
Are you suggesting the same technique as Erik? What do you mean by a truly back-facing edge?
@Kryzon, you said:
Fortunately, after some research I got to know the "Reverse-Extruded Shadow Volumes" technique by Renaldas Zioma.
In simple terms, to create the volume, instead of extruding the back faces and leaving the front faces in place, you do the opposite.
I fail to see how extruding the front-faces instead makes the self-shadowing problem any better. Don't have time to read through and digest a paper right now. Could you give me a few more details on how they resolve the self-shadowing issue? I already have ambient shadows in place. (which I assume means, only ambient light is rendered in shadowed pixels)
Pretty much did exactly what you said. Definitely fixes the self shadowing issue. However, the shadow shrinking produces a shadow with a jagged, per-polygon edge. So ironically it really just moves the same jagged artifacts off the self-shadows and puts them on the regular shadows. :\
Removing lighting from any edges that touches the shadow instead is an alternative, to grow the shaded part on the caster instead of shrinking the volume, but it's usually a bit more expensive to implement, depending on what rendering techniques you use..
For convex models like the sphere the easiest alternative is of course to not use self-shadowing on the caster, but that won't work on concave models where you want self-shadowing.
Another alternative, since the problem only occurs on the shadow caster, is to fix it by adding the lighting on edge polys on the caster, emulating shrinking the volume only on the caster. Run an extra lighting-pass on the polygons that are at the shadow edge, selected using the same algorithm as when constructing the shadow volumes (geometry-shader?).
So basically draw the model triangles for the caster, as for a lighting-pass, but without stencil test, and with a geometry shader that only outputs the triangles where the triangle-normal faces away from the light, and at least one vertex-normal faces the light. This will discard all polys except those that you know are in the shadow volume but have at least one normal that is facing the light. Should be relatively cheap.
It all depends on what artifacts or deviations from "correct" shadows are acceptable. The problem with smooth normals and exact silhouette shadows is that the silhouette is an approximation of a smooth surface, and the interpolated normals attempts to hide that fact. For perfectly hard shadows, the jagged edge on the sphere caster is actually correct.
Another way that might solve some things is by shrinking the silhouette inwards by moving the vertices toward the center instead of removing vertices, which in some circumstances can probably solve the problem, but of course has the potential to bring new artifacts.