Disclaimer: this is older OpenGL, and while I would love to switch to all shaders (not sure if that would be appropriate here even), I'd rather complete what I have instead of switching the whole engine over at this point. It is an older project, but I'd still like to see it wrapped up in its current form.
My project is a 3d platformer, you have cubes jumping around on other cubes and they use shadow volumes to show where they are in relation to the ground. Recently I added crumbling blocks, the kind that disappear a few seconds after you touch them. That's great! I have them set so their alpha is reduced based on how long it's been since you've touched them so they fade away after a few seconds. The problem is: even at alpha zero, they're affecting the depth test for my stencil buffer.
The white/grey blocks are the crumbling ones:
(Don't mind the missing face, it's part of an efficiency measure to remove unused faces that doesn't consider disappearing blocks yet)
The problem you see here is that the shadows, while drawn fine on the left and right, are not appearing in the middle behind the player. That's because there are crumbling blocks there, though they have 0% opacity/alpha. You can even see the outline of the cube just above and to the right of the player because that's the shape cut out of the shadows.
Also: the player automatically gets a silhouette when behind another object, same stencil style test as the shadows, so that's why the player is mostly yellow and shadowed yellow but then turns green at the top. It's explained here.
I don't want to leave anything out I might be missing, so here's the whole code chunk (you can see my comments trying to remember what I wrote... I've gotten better at this since 2012... ):
// Give shadows to everything!
void drawAllShadows(int player) {
// Blend function not causing any differences if enabled/disabled?
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
// GL_BLEND is the difference between black shadows and dark versions of the colors underneath
glEnable( GL_BLEND );
// Not sure GL_ALPHA is doing anything, nor ALPHA_TEST
glEnable( GL_ALPHA );
glEnable( GL_ALPHA_TEST );
// Color Mask with everything disabled means it doesn't use this data to draw real shapes/objects
// instead, we're going to use it for a stencil test later
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
// No Depth Test lets shadows overlap instead of their not-drawn sections deleting shadows behind them
glDepthMask(GL_FALSE);
// GL_STENCIL_TEST limits the shadows to being drawn in the stencils, which are the levelShadows
glEnable(GL_STENCIL_TEST);
// Just draw the level shadows as regular polygons, but with the stencil option
if (levelShadows) {
glEnableClientState(GL_COLOR_ARRAY);
glEnableClientState(GL_VERTEX_ARRAY);
// specify pointer to vertex array
glColorPointer(3, GL_FLOAT, 0, shadowColors);
glVertexPointer(3, GL_FLOAT, 0, shadowVertices);
glPushMatrix();
//glScalef(0.99,100.0,0.99);
//glTranslatef(0.0,-50.505,0.0);
glCullFace(GL_FRONT);
glStencilFunc(GL_ALWAYS, 0x0, 0xff);
// if depth fails (can't see only due to z) then increment pixel's value
glStencilOp(GL_KEEP, GL_INCR, GL_KEEP);
glDrawElements(GL_TRIANGLES, 36*shadowsVisible, GL_UNSIGNED_INT, shadowIndices);
glCullFace(GL_BACK);
glStencilFunc(GL_ALWAYS, 0x0, 0xff);
// if depth also fails for back, decrement it so it stays at zero
// but if one increments and the other does not...
// (we'll come back to that later when we see where to place shadows)
glStencilOp(GL_KEEP, GL_DECR, GL_KEEP);
glDrawElements(GL_TRIANGLES, 36*shadowsVisible, GL_UNSIGNED_INT, shadowIndices);
glPopMatrix();
// deactivate vertex arrays after drawing
glDisableClientState(GL_VERTEX_ARRAY);
glDisableClientState(GL_COLOR_ARRAY);
}
/*
for (int i=0; i<cubeNum; i++) {
// cubeWithinPlayerRange demonstrates noticeable improvements,
// castle level with 4player, sfx and music went from 7fps to 12.
// now goes from 24 to 35fps! Wow!
if (cubeWithinPlayerRange(i,player)) {
drawCubeShadow(i);
}
}*/
for (int i=0; i<cubiorNum; i++) { drawPlayerShadow(i); }
drawGoalShadow();
drawItemShadows();
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
// Where the front and the back are not the same (so front face could be seen but back was obscured)
// that is where a shadow is on the ground
// or that is what I believe NOTEQUAL and REPLACE are being used for
// all three replaces means no matter what (fail/zfail/succeed), we will the pixel with the new color
glStencilFunc(GL_NOTEQUAL, 0x0, 0xff);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
fillScreenWithShadow();
glDisable(GL_STENCIL_TEST);
glDisable( GL_BLEND );
// end of shadow stuff
}
// 100% copied from draw_shadow for testing purposes
// full permission to copy this though, according to Josh Beam:
// http://joshbeam.com/articles/stenciled_shadow_volumes_in_opengl/
void fillScreenWithShadow() {
// Fills the stencilled area with a shadow of 50% opacity on top of everything else
glPushMatrix();
glLoadIdentity();
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();
glOrtho(0, 1, 1, 0, 0, 1);
glDisable(GL_DEPTH_TEST);
glColor4f(0.0f, 0.0f, 0.0f, 0.5f);
glBegin(GL_QUADS);
glVertex2i(0, 0);
glVertex2i(0, 1);
glVertex2i(1, 1);
glVertex2i(1, 0);
glEnd();
glEnable(GL_DEPTH_TEST);
glPopMatrix();
glMatrixMode(GL_MODELVIEW);
glPopMatrix();
}
The function drawItemShadows is the exact one for the crumbling block shadows, but it works the same way as what you see here already. It increments the stencil's pixel value for front faces and decrements it for back faces, so it should be zero if both faces are visible or both are obscured, but if only one is then that's where a shadow lands. But again, the problem is that the z test there isn't taking into account transparent polygons. Before this code is where I draw the items and world itself (shadows are drawn last) and those are drawn with 3 point polygon indexes and 4 point color indexes, 4th slot being alpha.
So how do I skip alpha in my depth tests, but then apply that alpha to the shadow I draw ultimately? Is that possible, or do I need to break this up into separate passes somehow?
