Jump to content
  • Advertisement

2D Using a shader for a simple alpha mask/stencil

This topic is 518 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

So this is a question that I asked over on GameDev Stack Exchange, and got no love here:

In XNA I have 2 RenderTarget2D's, each with alpha layers, each with identical resolutions, and a surface type of Color. One is the base texture, and the other is the texture I want to use to further mask the base texture. I'm trying this out with a common stencil shader that's been tossed around on gamedev.stackexchange that looks like this:

sampler BaseTexture : register(s0);
sampler MaskTexture : register(s1) {
    addressU = Clamp;
    addressV = Clamp;

float MaskLocationX;
float MaskLocationY;
float MaskWidth;
float MaskHeight;
float BaseTextureLocationX;
float BaseTextureLocationY;
float BaseTextureWidth;
float BaseTextureHeight;

float4 StencilFunction(float2 texCoord : TEXCOORD0) : COLOR0 {
    float maskPixelX = texCoord.x * BaseTextureWidth + BaseTextureLocationX;
    float maskPixelY = texCoord.y * BaseTextureHeight + BaseTextureLocationY;
    float2 maskCoord = float2((maskPixelX - MaskLocationX) / MaskWidth, (maskPixelY - MaskLocationY) / MaskHeight);
    float4 bitMask = tex2D(MaskTexture, maskCoord);
    float4 tex = tex2D(BaseTexture, texCoord);
    return tex * (bitMask.a);

technique RenderStencil {
    pass P0 {
        PixelShader = compile ps_2_0 StencilFunction();

Technically, I'm using FNA, and the shader has been compiled (presumably successfully) using fxc.  It may also be relevant that I'm using JetBrains Rider under Linux instead of VS, though from what I can tell, this shouldn't be a problem.

I just want to apply this shader to these textures onto a 3rd render target.

I've tried a number of things, but this seems to be the only thing that causes some kind of response:

Manager.Device.Textures[0] = group[0].TempTarget;
Manager.Device.Textures[1] = group[1].TempTarget;
sceneBatch.Begin(SpriteSortMode.Immediate, BlendState.AlphaBlend, null, null, null, Map.StencilEffect);
sceneBatch.Draw(group[0].TempTarget, sceneRectangle, Color.White);
sceneBatch.Draw(group[1].TempTarget, sceneRectangle, Color.White);

group[0] and group[1] are my layer objects, and TempTarget is a RenderTarget2D containing the result of each layer being rendered flat.  I've tried a whole mess of different combinations of things, and either the 2 layers render unmodified, or not at all.  It seems no shading is taking place.

It's also likely that my shader could be majorly simplified.  Both incoming rendertargets will always be the same size.  I'm only just digging into shaders, but I want to get something working before I optimize.

I don't know if the draws are necessary. I don't know if I pass the base texture or the mask texture into Device.Texture[0] or Device.Texture[1]. And I don't know which if any texture needs to Draw() to the batch.

What's wrong? What am I missing? And why is it so difficult to find an answer to this in the vast wide Google?  Pixel shaders in XNA seem to be a very closely guarded secret.

Share this post

Link to post
Share on other sites

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!