Task: Mask off certain areas in a texture depending on a color key in an overlay texture (multi textured). The idea behind it is that certain overlays have holes that should punch through the base texture (grates sitting on a wall e.g.).
I have tried to put this in a shader combining base and overlay textures, with a mask texture telling which areas to leave transparent:
Fragment shader:
uniform sampler2D btmTex, topTex, maskTex;
uniform float grAlpha; //global alpha
vec4 topColor, btmColor;
float bMask; //could do without an extra variable, but for the sake of clarity ...
void main (void)
{
bMask = texture2D (maskTex, vec2 (gl_TexCoord [2])).a;
if (bMask < 0.5)
discard;
else {
topColor = texture2D (topTex, vec2(gl_TexCoord [1]));
btmColor = texture2D (btmTex, vec2(gl_TexCoord [0]));
if(topColor.a == 0.0)
gl_FragColor=vec4(vec3(btmColor),btmColor.a*grAlpha)*gl_Color;
else
gl_FragColor = vec4 (vec3 (mix (btmColor, topColor, topColor.a)), (btmColor.a + topColor.a) * grAlpha) * gl_Color;
}
}
Vertex shader:
void main (void)
{
gl_TexCoord [0] = gl_MultiTexCoord0;
gl_TexCoord [1] = gl_MultiTexCoord1;
gl_TexCoord [2] = gl_MultiTexCoord2;
gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
gl_FrontColor = gl_Color;
}
The mask texture has the format GL_ALPHA and the source format GL_UNSIGNED_BYTE. A value of 255 (1) means no hole, 0 means hole. Tried GL_RGBA4 too, but didn't work either.
bmTex is bound to GL_TEXTURE0, topTex to GL_TEXTURE1, maskTex to GL_TEXTURE2 (my gfx card has enough texture units).
To test it I have set the entire mask texture to "0". The shader should mask the entire texture off then. It doesn't though.
Simple enough, one should think. The shader doesn't work though. It does something, because if I have it unconditionally discard every fragment, nothing gets rendered. Same for "if (bmMask > 0.5) discard;"
I have a similar shader only working with bmTex and ovlTex and directly checking for the color key, and it works. As the color key gets messed up by the texture filtering before arriving in the shader though, the result doesn't look that good, so I would like to use a mask texture.
What the heck am I doing wrong? I have no clue.
[Edited by - karx11erx on October 11, 2007 5:00:46 PM]
