Jump to content
  • Advertisement
JUr

2D Using a shader for a simple alpha mask/stencil

This topic is 374 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

So this is a question that I asked over on GameDev Stack Exchange, and got no love here:

In XNA I have 2 RenderTarget2D's, each with alpha layers, each with identical resolutions, and a surface type of Color. One is the base texture, and the other is the texture I want to use to further mask the base texture. I'm trying this out with a common stencil shader that's been tossed around on gamedev.stackexchange that looks like this:

sampler BaseTexture : register(s0);
sampler MaskTexture : register(s1) {
    addressU = Clamp;
    addressV = Clamp;
};

float MaskLocationX;
float MaskLocationY;
float MaskWidth;
float MaskHeight;
float BaseTextureLocationX;
float BaseTextureLocationY;
float BaseTextureWidth;
float BaseTextureHeight;

float4 StencilFunction(float2 texCoord : TEXCOORD0) : COLOR0 {
    float maskPixelX = texCoord.x * BaseTextureWidth + BaseTextureLocationX;
    float maskPixelY = texCoord.y * BaseTextureHeight + BaseTextureLocationY;
    float2 maskCoord = float2((maskPixelX - MaskLocationX) / MaskWidth, (maskPixelY - MaskLocationY) / MaskHeight);
    float4 bitMask = tex2D(MaskTexture, maskCoord);
    float4 tex = tex2D(BaseTexture, texCoord);
    return tex * (bitMask.a);
}

technique RenderStencil {
    pass P0 {
        PixelShader = compile ps_2_0 StencilFunction();
    }
}

Technically, I'm using FNA, and the shader has been compiled (presumably successfully) using fxc.  It may also be relevant that I'm using JetBrains Rider under Linux instead of VS, though from what I can tell, this shouldn't be a problem.

I just want to apply this shader to these textures onto a 3rd render target.

I've tried a number of things, but this seems to be the only thing that causes some kind of response:

Manager.Device.Textures[0] = group[0].TempTarget;
Manager.Device.Textures[1] = group[1].TempTarget;
sceneBatch.Begin(SpriteSortMode.Immediate, BlendState.AlphaBlend, null, null, null, Map.StencilEffect);
sceneBatch.Draw(group[0].TempTarget, sceneRectangle, Color.White);
sceneBatch.Draw(group[1].TempTarget, sceneRectangle, Color.White);
sceneBatch.End();

group[0] and group[1] are my layer objects, and TempTarget is a RenderTarget2D containing the result of each layer being rendered flat.  I've tried a whole mess of different combinations of things, and either the 2 layers render unmodified, or not at all.  It seems no shading is taking place.

It's also likely that my shader could be majorly simplified.  Both incoming rendertargets will always be the same size.  I'm only just digging into shaders, but I want to get something working before I optimize.

I don't know if the draws are necessary. I don't know if I pass the base texture or the mask texture into Device.Texture[0] or Device.Texture[1]. And I don't know which if any texture needs to Draw() to the batch.

What's wrong? What am I missing? And why is it so difficult to find an answer to this in the vast wide Google?  Pixel shaders in XNA seem to be a very closely guarded secret.

Share this post


Link to post
Share on other sites
Advertisement

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!