I am trying to get volume shadows to work. My degenerate mesh is correct, I have rendered it to a framebuffer and saw that it was good.
But when I try to get it on a stencil buffer, my stencil buffer stays blank or has junk.

First I render my normal scene, I make sure my stencil buffer is cleared, and the Z-buffer is filled with correct values.
Then I enable the stencil buffer (I use double sided), I disable the pixel mask, and I draw the degenerate meshes.
I enable the pixel mask and draw shadow quad.

It doesn't work, on my windows machine, there is no shadow, on my mac, there is junk on the screen (like it has been raining shadow drops on my camera). To make sure that the problem is with the stencil buffer, I have rendered out a fully white quad to a framebuffer with the stencil enabled, and I log that framebuffer. On my windows, it's completely white, on my mac, it has junk.

So I am quite sure that I am doing something wrong with the stenciing. It must be some kind of initialisation issue or something, but I follow every rule in the blue book and several tutorials I found.

Here's the code that does this:

glClearColor(0, 0, 0, 0);
glClearDepth(1.0f);
glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glEnable(GL_DEPTH_TEST);
glEnable(GL_CULL_FACE);

... render scene ...

glEnable(GL_DEPTH_TEST);
glDisable(GL_CULL_FACE);
glDepthMask(GL_FALSE);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

glEnable(GL_STENCIL_TEST);
glStencilOpSeparate(GL_BACK, GL_KEEP, GL_DECR, GL_KEEP);
glStencilOpSeparate(GL_FRONT, GL_KEEP, GL_INCR, GL_KEEP);
glStencilMask(0xff);
glStencilFunc(GL_ALWAYS, 0x01, 0xff);

... render degenerate shadow mesh ...

glBindFramebuffer(GL_FRAMEBUFFER, m_fb_test);
glDepthMask(GL_TRUE);
glDisable(GL_DEPTH_TEST);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glStencilFunc(GL_LEQUAL, 0x01, 0xff);
shader = m_shader_flatfill;
glUseProgram(shader);
GLuint aVertexPosition = glGetAttribLocation(shader, "aVertexPosition");
glEnableVertexAttribArray(aVertexPosition);
glBindBuffer(GL_ARRAY_BUFFER, m_quad);
glVertexAttribPointer(aVertexPosition, 2, GL_FLOAT, false, 4 * sizeof(float), (GLvoid*)(0)); // the 4 * is correct, the vertex buffer also has texture coordinates that are not used by the shader
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glDisableVertexAttribArray(aVertexPosition);
glDisable(GL_STENCIL_TEST);

=> m_fb_test contains white or junk

I have been doing some testing later on the evening and I had used the screen as render target for the stencil buffers => test buffer, it looks like my stencil buffer is ANIMATING with junk???? But I clear it every frame, so I don't understand.

Could this have to do with how I initialize my opengl? Or am I just doing something wrong with my stencil buffer?

Initialization of opengl:

static PIXELFORMATDESCRIPTOR pfd =
{
sizeof(PIXELFORMATDESCRIPTOR), // Size Of This Pixel Format Descriptor
1, // Version Number
PFD_DRAW_TO_WINDOW | // Format Must Support Window
PFD_SUPPORT_OPENGL | // Format Must Support OpenGL
PFD_DOUBLEBUFFER, // Must Support Double Buffering
PFD_TYPE_RGBA, // Request An RGBA Format
32, // Select Our Color Depth
0, 0, 0, 0, 0, 0, // Color Bits Ignored
0, // No Alpha Buffer
0, // Shift Bit Ignored
0, // No Accumulation Buffer
0, 0, 0, 0, // Accumulation Bits Ignored
32, // 32Bit Z-Buffer (Depth Buffer)
1, // Stencil Buffer
0, // No Auxiliary Buffer
PFD_MAIN_PLANE, // Main Drawing Layer
0, // Reserved
0, 0, 0 // Layer Masks Ignored
};

I have tried putting 24 in z-buffer depth and 8 in stencil buffer bits, no help at all, but it shouldn't as I use a different framebuffer to render to, and that framebuffer has a DEPTH24_STENCIL8 attached.

Anyone?

I found out what the problem was:

I made the false assumption that when attaching a DEPTH24_STENCIL8 buffer to GL_DEPTH_ATTACHMENT to a framebuffer would be enough to make it also have a stencil buffer.
But when I queried for GL_STENCIL_BITS, I found out that I had 0 stencil bits.

So, you also need to attach the same buffer as GL_STENCIL_ATTACHMENT.

Problem solved!

Well, GL_STENCIL_BITS is for querying the framebuffer stencil. You seem to be talking about attaching a depth_stencil buffer to a FBO, which is not the same thing. The main framebuffer is not setup with the same API as FBO (which are a later addition, they are 2 years old now).The main framebuffer is OS specific (GDI function cal = SetPixelFormat).

Yes, I know. But when everything fails, you begin to doubt what you know.

Actually, my engine only needs the depth_stencil buffer in framebuffers, not in the main backbuffer, so I will disable it there.