# Simple Stencil Buffer Operations

This topic is 2610 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Dear GameDev,

Graphics can be hard to debug. Even harder when you can't see what you're doing. I'm trying to get some simple stencil buffer functionality working - but so far I've not had that much luck. So instead of fumbling around in the dark, I decided to try and read what the stencil buffer contains. However - even this is proving to be more than a match for me. I google'd around and found someone in a similar fix. But I've tried implementing the code that he suggests

 unsigned char stencilData[] = { 9, 8, 7, 6, 5, 4, 3, 2, 1, }; glEnable(GL_STENCIL_TEST); glPixelStorei(GL_UNPACK_ALIGNMENT, 1); glPixelStorei(GL_UNPACK_ROW_LENGTH, 3); glPixelStorei(GL_UNPACK_SKIP_ROWS, 0); glPixelStorei(GL_UNPACK_SKIP_PIXELS, 0); glRasterPos2i(0, 0); glDrawPixels(3, 3, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, stencilData); unsigned char *data = new unsigned char[9]; glPixelStorei(GL_PACK_ALIGNMENT, 4); glPixelStorei(GL_PACK_ROW_LENGTH, 3); glPixelStorei(GL_PACK_SKIP_ROWS, 0); glPixelStorei(GL_PACK_SKIP_PIXELS, 0); glReadPixels(0, 0, 3, 3, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, data); for (int y = 0; y < 3; y++) { for (int x = 0; x < 3; x++) { std::cout << (int)data[y * 3 + x] << "\t"; } std::cout << std::endl; } 

without any luck. As far as I can tell - the glPixelStorei calls are redundant. I'd think that reading and writing to the stencil buffers would not require meddling with those settings. I get a similar result to the one he posts initially. My result is:

0 0 0
186 0 0
0 186 0

In a follow up post the user writes...

[color="#1C2837"]:-| Find out that i forgot to Enable Stencil test before render, and after change the
"glPixelStorei(GL_UNPACK_ALIGNMENT, 4);"
to
"glPixelStorei(GL_UNPACK_ALIGNMENT, 1);"

[/quote]

I fixed the GL_UNPACK alignment - as the above code shows - but that corresponds to the default values unless I am mistaken. However, he writes he forgot to enable the stencil test before rendering. Isnt the following call:
glEnable(GL_STENCIL_TEST);
enough to enable stencil test?

I'm confused. I hope someone has an idea as to what I'm doing wrong...

Regards,
Gazoo

##### Share on other sites
Why do you want to enable the stencil test when you want to write to the stencil buffer with glDrawPixels and then read from the stencil buffer?

I'm confused. I hope someone has an idea as to what I'm doing wrong...[/quote]

Well, think about what it means to enable the stencil test. What happens when you don't setup glStencilFunc, glStencilMask, glStencilOpglStencilOp. They get left on defaults.
But still, why would you enable the stencil test if all you are doing is directly writing to it?

glPixelStorei(GL_PACK_ALIGNMENT, 4);
Why is your pack alignment 4 and yet
glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
unpack alignment 1?

##### Share on other sites
Hey V-Man,

Because I'm clearly a bit clueless. I thought that without enabling the "STENCIL_TEST" is was not possible to access the stencil buffer at all... I suppose I was mistaken?

My code looks as follows:

 glClear( GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT | GL_COLOR_BUFFER_BIT ); unsigned char stencilData[] = { 9, 8, 7, 6, 5, 4, 3, 2, 1, }; //glEnable(GL_STENCIL_TEST); glPixelStorei(GL_UNPACK_ALIGNMENT, 1); glPixelStorei(GL_UNPACK_ROW_LENGTH, 3); glPixelStorei(GL_UNPACK_SKIP_ROWS, 0); glPixelStorei(GL_UNPACK_SKIP_PIXELS, 0); glRasterPos2i(0, 0); glDrawPixels(3, 3, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, stencilData); unsigned char *data = new unsigned char[9]; glPixelStorei(GL_PACK_ALIGNMENT, 1); glPixelStorei(GL_PACK_ROW_LENGTH, 3); glPixelStorei(GL_PACK_SKIP_ROWS, 0); glPixelStorei(GL_PACK_SKIP_PIXELS, 0); glReadPixels(0, 0, 3, 3, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, data); for (int y = 0; y < 3; y++) { for (int x = 0; x < 3; x++) { std::cout << (int)data[y * 3 + x] << "\t"; } std::cout << std::endl; } 

I read the link you provided and figured it was best to keep it simple and go with an alignment of 1. However, I still get the wrong values. With the above code I get all 0's back when reading the stencil buffer. I am still unsure if I am not writing to it properly or if I am not reading it right... Or both...

Regards,
Gazoo

##### Share on other sites
You can probably leave GL_UNPACK_ROW_LENGTH and GL_PACK_ROW_LENGTH to their default values, 0.
Since the raster position gets transformed by the modelview matrix, perhaps you should set the modelview to identity.To test, write something visible like BGRA values to the screen. If the BGRA values show up, then check to make sure there is a stencil buffer. glGetIntegerv(GL_STENCIL_BITS, &integer) will give you that.

##### Share on other sites
It took me quite a while to figure out. I had to go all the way back to just trying to make the whole screen red, read a little, then make a smaller piece red, read some more, then switch to unsigned bytes, then try again, then... and so forth...

Anyway - it turns out the devil was the rasterpos call:

glRasterPos2i

I think it would probably work if I had an orthogonal projection running, but I've just stuck with the defaults seeing as I am overwriting all the color on the screen anyway. So in short - you don't need that call if you're leaving OpenGL at its defaults - in fact - using it will screw things up. Instead, either use the following call:

glWindowPos2i

which takes care of the whole world space to screen space conversion, or just don't call anything. The initial reading position is just fine where it is if you've left your openGL settings at relatively default values.

I can read and write to the stencil buffer just fine now!

Thanks for the help V-Man...

P.s oh yea, almost forgot - you do need to set the pack alignment to 1 [color=#1C2837][font=CourierNew, monospace][size=2][color=#000000]glPixelStorei[color=#666600]([color=#000000]GL_PACK_ALIGNMENT[color=#666600],[color=#000000] [color=#006666]1[color=#666600]);[/font]... Leave the rest at their defaults...

Here's mah code:

 void mds_classic::debugRegularBuffer() { glClear( GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT | GL_COLOR_BUFFER_BIT ); glLoadIdentity(); unsigned char *pixData = new unsigned char[20]; unsigned long pixDataPtr = 0; for (unsigned int i = 0; i < 20; ++i) { pixData[pixDataPtr] = 123; ++pixDataPtr; //pixData[pixDataPtr] = 0; ++pixDataPtr; //pixData[pixDataPtr] = 180; ++pixDataPtr; //pixData[pixDataPtr] = 255; ++pixDataPtr; } glDrawPixels(4, 5, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, pixData); //glRasterPos2i(0, 0); //glDrawPixels(2, 2, GL_RGBA, GL_UNSIGNED_BYTE, bufferData); //unsigned char *data = new unsigned char[16]; //if (runonce) { unsigned char extData[25]; glPixelStorei(GL_PACK_ALIGNMENT, 1); //glPixelStorei(GL_PACK_ROW_LENGTH, 3); //glReadBuffer(GL_FRONT); glReadPixels(0, 0, 5, 5, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, extData); imdebug("lum b=8 w=%d h=%d t=%s %p", 5, 5, "Debugging Buffer Data", extData); //runonce = false; //} //imdebug("rgba b=8 w=%d h=%d t=%s %p", 2, 2, "Debugging Buffer Data", data); delete [] pixData; } 

1. 1
2. 2
Rutin
24
3. 3
4. 4
JoeJ
18
5. 5

• 14
• 22
• 11
• 11
• 9
• ### Forum Statistics

• Total Topics
631766
• Total Posts
3002215
×