Jump to content
  • Advertisement
Sign in to follow this  
scippie

OpenGL Can't get stencil buffer to work

This topic is 2553 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I am trying to get volume shadows to work. My degenerate mesh is correct, I have rendered it to a framebuffer and saw that it was good.
But when I try to get it on a stencil buffer, my stencil buffer stays blank or has junk.

First I render my normal scene, I make sure my stencil buffer is cleared, and the Z-buffer is filled with correct values.
Then I enable the stencil buffer (I use double sided), I disable the pixel mask, and I draw the degenerate meshes.
I enable the pixel mask and draw shadow quad.

It doesn't work, on my windows machine, there is no shadow, on my mac, there is junk on the screen (like it has been raining shadow drops on my camera). To make sure that the problem is with the stencil buffer, I have rendered out a fully white quad to a framebuffer with the stencil enabled, and I log that framebuffer. On my windows, it's completely white, on my mac, it has junk.

So I am quite sure that I am doing something wrong with the stenciing. It must be some kind of initialisation issue or something, but I follow every rule in the blue book and several tutorials I found.

Here's the code that does this:

glClearColor(0, 0, 0, 0);
glClearDepth(1.0f);
glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glEnable(GL_DEPTH_TEST);
glEnable(GL_CULL_FACE);

... render scene ...

glEnable(GL_DEPTH_TEST);
glDisable(GL_CULL_FACE);
glDepthMask(GL_FALSE);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

glEnable(GL_STENCIL_TEST);
glStencilOpSeparate(GL_BACK, GL_KEEP, GL_DECR, GL_KEEP);
glStencilOpSeparate(GL_FRONT, GL_KEEP, GL_INCR, GL_KEEP);
glStencilMask(0xff);
glStencilFunc(GL_ALWAYS, 0x01, 0xff);

... render degenerate shadow mesh ...

glBindFramebuffer(GL_FRAMEBUFFER, m_fb_test);
glDepthMask(GL_TRUE);
glDisable(GL_DEPTH_TEST);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glStencilFunc(GL_LEQUAL, 0x01, 0xff);
shader = m_shader_flatfill;
glUseProgram(shader);
GLuint aVertexPosition = glGetAttribLocation(shader, "aVertexPosition");
glEnableVertexAttribArray(aVertexPosition);
glBindBuffer(GL_ARRAY_BUFFER, m_quad);
glVertexAttribPointer(aVertexPosition, 2, GL_FLOAT, false, 4 * sizeof(float), (GLvoid*)(0)); // the 4 * is correct, the vertex buffer also has texture coordinates that are not used by the shader
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glDisableVertexAttribArray(aVertexPosition);
glDisable(GL_STENCIL_TEST);

=> m_fb_test contains white or junk


I have been doing some testing later on the evening and I had used the screen as render target for the stencil buffers => test buffer, it looks like my stencil buffer is ANIMATING with junk???? But I clear it every frame, so I don't understand.

Could this have to do with how I initialize my opengl? Or am I just doing something wrong with my stencil buffer?

Initialization of opengl:

static PIXELFORMATDESCRIPTOR pfd =
{
sizeof(PIXELFORMATDESCRIPTOR), // Size Of This Pixel Format Descriptor
1, // Version Number
PFD_DRAW_TO_WINDOW | // Format Must Support Window
PFD_SUPPORT_OPENGL | // Format Must Support OpenGL
PFD_DOUBLEBUFFER, // Must Support Double Buffering
PFD_TYPE_RGBA, // Request An RGBA Format
32, // Select Our Color Depth
0, 0, 0, 0, 0, 0, // Color Bits Ignored
0, // No Alpha Buffer
0, // Shift Bit Ignored
0, // No Accumulation Buffer
0, 0, 0, 0, // Accumulation Bits Ignored
32, // 32Bit Z-Buffer (Depth Buffer)
1, // Stencil Buffer
0, // No Auxiliary Buffer
PFD_MAIN_PLANE, // Main Drawing Layer
0, // Reserved
0, 0, 0 // Layer Masks Ignored
};


I have tried putting 24 in z-buffer depth and 8 in stencil buffer bits, no help at all, but it shouldn't as I use a different framebuffer to render to, and that framebuffer has a DEPTH24_STENCIL8 attached.

Anyone?

Share this post


Link to post
Share on other sites
Advertisement
I found out what the problem was:

I made the false assumption that when attaching a DEPTH24_STENCIL8 buffer to GL_DEPTH_ATTACHMENT to a framebuffer would be enough to make it also have a stencil buffer.
But when I queried for GL_STENCIL_BITS, I found out that I had 0 stencil bits.

So, you also need to attach the same buffer as GL_STENCIL_ATTACHMENT.

Problem solved!

Share this post


Link to post
Share on other sites

I found out what the problem was:

I made the false assumption that when attaching a DEPTH24_STENCIL8 buffer to GL_DEPTH_ATTACHMENT to a framebuffer would be enough to make it also have a stencil buffer.
But when I queried for GL_STENCIL_BITS, I found out that I had 0 stencil bits.

So, you also need to attach the same buffer as GL_STENCIL_ATTACHMENT.

Problem solved!


Well, GL_STENCIL_BITS is for querying the framebuffer stencil. You seem to be talking about attaching a depth_stencil buffer to a FBO, which is not the same thing. The main framebuffer is not setup with the same API as FBO (which are a later addition, they are 2 years old now).The main framebuffer is OS specific (GDI function cal = SetPixelFormat).

Share this post


Link to post
Share on other sites

Well, GL_STENCIL_BITS is for querying the framebuffer stencil. You seem to be talking about attaching a depth_stencil buffer to a FBO, which is not the same thing. The main framebuffer is not setup with the same API as FBO (which are a later addition, they are 2 years old now).The main framebuffer is OS specific (GDI function cal = SetPixelFormat).


Yes, I know. But when everything fails, you begin to doubt what you know.

Actually, my engine only needs the depth_stencil buffer in framebuffers, not in the main backbuffer, so I will disable it there.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!