Dear GameDev,

Graphics can be hard to debug. Even harder when you can't see what you're doing. I'm trying to get some simple stencil buffer functionality working - but so far I've not had that much luck. So instead of fumbling around in the dark, I decided to try and read what the stencil buffer contains. However - even this is proving to be more than a match for me. I google'd around and found someone in a similar fix. But I've tried implementing the code that he suggests


unsigned char stencilData[] =
{
9, 8, 7,
6, 5, 4,
3, 2, 1,
};
glEnable(GL_STENCIL_TEST);

glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
glPixelStorei(GL_UNPACK_ROW_LENGTH, 3);
glPixelStorei(GL_UNPACK_SKIP_ROWS, 0);
glPixelStorei(GL_UNPACK_SKIP_PIXELS, 0);

glRasterPos2i(0, 0);
glDrawPixels(3, 3, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, stencilData);

unsigned char *data = new unsigned char[9];
glPixelStorei(GL_PACK_ALIGNMENT, 4);
glPixelStorei(GL_PACK_ROW_LENGTH, 3);
glPixelStorei(GL_PACK_SKIP_ROWS, 0);
glPixelStorei(GL_PACK_SKIP_PIXELS, 0);

glReadPixels(0, 0, 3, 3, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, data);
for (int y = 0; y < 3; y++) {
for (int x = 0; x < 3; x++) {
std::cout << (int)data[y * 3 + x] << "\t";
}
std::cout << std::endl;
}



without any luck. As far as I can tell - the glPixelStorei calls are redundant. I'd think that reading and writing to the stencil buffers would not require meddling with those settings. I get a similar result to the one he posts initially. My result is:

0 0 0
186 0 0
0 186 0

In a follow up post the user writes...

[color="#1C2837"]:-| Find out that i forgot to Enable Stencil test before render, and after change the
"glPixelStorei(GL_UNPACK_ALIGNMENT, 4);"
to
"glPixelStorei(GL_UNPACK_ALIGNMENT, 1);"

Reading the stencil buffer with glReadPixels works...
[/quote]

I fixed the GL_UNPACK alignment - as the above code shows - but that corresponds to the default values unless I am mistaken. However, he writes he forgot to enable the stencil test before rendering. Isnt the following call:
glEnable(GL_STENCIL_TEST);
enough to enable stencil test?

I'm confused. I hope someone has an idea as to what I'm doing wrong...

Regards,
Gazoo

Hey V-Man,

Because I'm clearly a bit clueless. I thought that without enabling the "STENCIL_TEST" is was not possible to access the stencil buffer at all... I suppose I was mistaken?

My code looks as follows:


glClear( GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT | GL_COLOR_BUFFER_BIT );

unsigned char stencilData[] =
{
9, 8, 7,
6, 5, 4,
3, 2, 1,
};
//glEnable(GL_STENCIL_TEST);

glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
glPixelStorei(GL_UNPACK_ROW_LENGTH, 3);
glPixelStorei(GL_UNPACK_SKIP_ROWS, 0);
glPixelStorei(GL_UNPACK_SKIP_PIXELS, 0);

glRasterPos2i(0, 0);
glDrawPixels(3, 3, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, stencilData);

unsigned char *data = new unsigned char[9];
glPixelStorei(GL_PACK_ALIGNMENT, 1);
glPixelStorei(GL_PACK_ROW_LENGTH, 3);
glPixelStorei(GL_PACK_SKIP_ROWS, 0);
glPixelStorei(GL_PACK_SKIP_PIXELS, 0);

glReadPixels(0, 0, 3, 3, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, data);
for (int y = 0; y < 3; y++) {
for (int x = 0; x < 3; x++) {
std::cout << (int)data[y * 3 + x] << "\t";
}
std::cout << std::endl;
}



I read the link you provided and figured it was best to keep it simple and go with an alignment of 1. However, I still get the wrong values. With the above code I get all 0's back when reading the stencil buffer. I am still unsure if I am not writing to it properly or if I am not reading it right... Or both...

Regards,
Gazoo

It took me quite a while to figure out. I had to go all the way back to just trying to make the whole screen red, read a little, then make a smaller piece red, read some more, then switch to unsigned bytes, then try again, then... and so forth...

Anyway - it turns out the devil was the rasterpos call:

glRasterPos2i

I think it would probably work if I had an orthogonal projection running, but I've just stuck with the defaults seeing as I am overwriting all the color on the screen anyway. So in short - you don't need that call if you're leaving OpenGL at its defaults - in fact - using it will screw things up. Instead, either use the following call:

glWindowPos2i

which takes care of the whole world space to screen space conversion, or just don't call anything. The initial reading position is just fine where it is if you've left your openGL settings at relatively default values.

I can read and write to the stencil buffer just fine now!

Thanks for the help V-Man...

P.s oh yea, almost forgot - you do need to set the pack alignment to 1 [color=#1C2837][font=CourierNew, monospace][size=2][color=#000000]glPixelStorei[color=#666600]([color=#000000]GL_PACK_ALIGNMENT[color=#666600],[color=#000000] [color=#006666]1[color=#666600]);[/font]... Leave the rest at their defaults...

Here's mah code:



void mds_classic::debugRegularBuffer() {
glClear( GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT | GL_COLOR_BUFFER_BIT );
glLoadIdentity();

unsigned char *pixData = new unsigned char[20];

unsigned long pixDataPtr = 0;
for (unsigned int i = 0; i < 20; ++i) {
pixData[pixDataPtr] = 123; ++pixDataPtr;
//pixData[pixDataPtr] = 0; ++pixDataPtr;
//pixData[pixDataPtr] = 180; ++pixDataPtr;
//pixData[pixDataPtr] = 255; ++pixDataPtr;
}

glDrawPixels(4, 5, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, pixData);

//glRasterPos2i(0, 0);
//glDrawPixels(2, 2, GL_RGBA, GL_UNSIGNED_BYTE, bufferData);

//unsigned char *data = new unsigned char[16];

//if (runonce) {
unsigned char extData[25];
glPixelStorei(GL_PACK_ALIGNMENT, 1);
//glPixelStorei(GL_PACK_ROW_LENGTH, 3);
//glReadBuffer(GL_FRONT);
glReadPixels(0, 0, 5, 5, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE, extData);
imdebug("lum b=8 w=%d h=%d t=%s %p", 5, 5, "Debugging Buffer Data", extData);
//runonce = false;
//}

//imdebug("rgba b=8 w=%d h=%d t=%s %p", 2, 2, "Debugging Buffer Data", data);

delete [] pixData;
}