I'm trying to figure out what the most efficient method is of applying an alpha transparency mask to a DirectX 9 texture in memory without using shaders. The current idea for implementing it is to do something like the following:


LPDIRECT3DTEXTURE9 texture, mask;

// Get the texture and the mask to apply from somewhere.

texture->Lock();
mask->Lock();

// Copy only the alpha channel from the mask into the texture, effectively applying the mask.
for (int32u i = 0; i < textureWIdth; ++i)
{
textureBits.alpha = maskBits.alpha;
}

texture->Unlock();
mask->Unlock();


As you can see, this is a rather brute force method that would end up being extremely slow on large images, but I don't know of anything in DirectX 9 for doing this besides using shaders. So my question to all of you: Are there any DirectX helper function that can be used to obtain the same results?

I'm not sure if this changes anything, but the texture I'm trying to apply the mask to is being used in off-screen rendering, so it is a render target.

This is possible in fixed function texture stages; sample the color texture in the first stage, and in the second stage, get the alpha from the second texture but take the color from the previous stage (D3DTA_CURRENT). The texture stage setup runs on the GPU hardware, unlike your proposed approach.

That sounds interesting, but I have no idea how to do something like that. Does this have to be done every frame before rendering or something? If you have any articles or anything on it or could post some pseudo code that would be very appreciated I'll try to do a bit of research on my own though.

A pixel shader would be more simple here, though

I know it would be, but the engine I'm using has no support for shaders whatsoever and it would be rather difficult to add it, especially with the time constraints on the project.

EDIT: Well I looked into it and I sort of understand what to do, but I can't figure out the mixture of texture stage states to get it working. This is what I have currently, which is resulting in no change:


device->SetTexture(0, fTexture);
device->SetTexture(1, fMaskTexture);

// Only use the color from the first texture.
device->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_MODULATE);
device->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_COLORARG2, D3DTA_DIFFUSE);
device->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
device->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);

// Then use the color from the previous texture, and the alpha from the mask.
device->SetTextureStageState(1, D3DTSS_COLOROP, D3DTOP_SELECTARG1);
device->SetTextureStageState(1, D3DTSS_COLORARG1, D3DTA_CURRENT);
device->SetTextureStageState(1, D3DTSS_COLORARG2, D3DTA_CURRENT);
device->SetTextureStageState(1, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
device->SetTextureStageState(1, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
device->SetTextureStageState(1, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);


Just rendering it to a primitive right below those calls with:


device->SetSamplerState(0, D3DSAMP_MAGFILTER, fD3DTextureFilterType);
device->SetSamplerState(0, D3DSAMP_MINFILTER, fD3DTextureFilterType);

device->SetStreamSource(0, fVertexBuffer, 0, sizeof(DXUVert_TEXTURED));

device->DrawPrimitive(D3DPT_TRIANGLEFAN, 0, 2);


Currently they are being set to:


device->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_SRCALPHA);
device->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA);
device->SetRenderState(D3DRS_ALPHABLENDENABLE, TRUE);


Maybe there's something wrong with the mask I'm using? Here are the images I'm testing with:

The main image to apply the mask to.
[attachment=9596:test.png]

The image mask to apply. It has a green color just to make it more obvious if there was a problem with the color part. The empty areas have zero alpha, which needs to be applied to the first image somehow.
[attachment=9597:testMask.png]

This is what the result should be.
[attachment=9598:testResult.png]

Well I sort of figured out what the problem is but I don't know how to fix it. After tinkering a bit I noticed that for some reason it's not picking up the alpha in the mask at all. If I set the states to:


// Combine the colors and use alpha from the mask only.
device->SetTextureStageState(1, D3DTSS_COLOROP, D3DTOP_MODULATE);
device->SetTextureStageState(1, D3DTSS_COLORARG1, D3DTA_CURRENT);
device->SetTextureStageState(1, D3DTSS_COLORARG2, D3DTA_TEXTURE);
device->SetTextureStageState(1, D3DTSS_ALPHAOP, D3DTOP_SELECTARG2);
device->SetTextureStageState(1, D3DTSS_ALPHAARG1, D3DTA_CURRENT);
device->SetTextureStageState(1, D3DTSS_ALPHAARG2, D3DTA_TEXTURE);


The texture gets tinted green as you would expect, except that the areas that are supposed to have zero alpha are also tinted green. Those areas should remain the same shouldn't they? The only logical reason why that would be happening would be because the alpha in the mask isn't getting applied or is getting ignored for some reason.

EDIT: After even more investigation I've narrowed it down to what the problem is. For some reason it's only using a single pixel, the top-left pixel at (0, 0), from the mask to apply to the entire texture. Anyone know why it would be doing this?

EDIT2: Think I figured it out! Took a lunch break to clear my head and realized something extremely important was mising; A second set of texture coordinates in the vertex data. I haven't tested it yet but I'm pretty sure this is why it isn't working. I'll post again once I confirm.

Well that did it, now it's working perfectly. Thanks a lot for you help Nik02. In case anyone else needs help here's some pseudo code for how to get this working:


// The vertex struct to use has two sets of texture coordinates, one for the main texture and one for the mask.
struct tDXVertexTextureMask
{
FLOAT x, y, z;
D3DCOLOR color;
FLOAT u1, v1;
FLOAT u2, v2;
};

// Define the FVF to use.
#define D3DFVF_TEXTURE_MASK (D3DFVF_XYZ | D3DFVF_DIFFUSE | D3DFVF_TEX2)

// Create your vertex buffer like usual using the above struct and FVF. When applying the texture coordinates,
// make sure to set u1 = u2 and v1 = v2, so the textures will line up exactly.

// Time to render! Setup the FVF and texture states.
device->SetFVF(D3DFVF_TEXTURE_MASK);

device->SetTexture(0, fTexture);
device->SetTexture(1, fMaskTexture);

device->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_MODULATE);
device->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_COLORARG2, D3DTA_DIFFUSE);
device->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
device->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);

// Use the color from the previous texture, and blend the alpha from the mask.
device->SetTextureStageState(1, D3DTSS_COLOROP, D3DTOP_SELECTARG1);
device->SetTextureStageState(1, D3DTSS_COLORARG1, D3DTA_CURRENT);
device->SetTextureStageState(1, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
device->SetTextureStageState(1, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
device->SetTextureStageState(1, D3DTSS_ALPHAARG2, D3DTA_CURRENT);

// Render your primative and voila! Masked texture!