This is possible in fixed function texture stages; sample the color texture in the first stage, and in the second stage, get the alpha from the second texture but take the color from the previous stage (D3DTA_CURRENT). The texture stage setup runs on the GPU hardware, unlike your proposed approach.

That sounds interesting, but I have no idea how to do something like that. Does this have to be done every frame before rendering or something? If you have any articles or anything on it or could post some pseudo code that would be very appreciated I'll try to do a bit of research on my own though.

A pixel shader would be more simple here, though

I know it would be, but the engine I'm using has no support for shaders whatsoever and it would be rather difficult to add it, especially with the time constraints on the project.

EDIT: Well I looked into it and I sort of understand what to do, but I can't figure out the mixture of texture stage states to get it working. This is what I have currently, which is resulting in no change:

device->SetTexture(0, fTexture);
device->SetTexture(1, fMaskTexture);

// Only use the color from the first texture.
device->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_MODULATE);
device->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_COLORARG2, D3DTA_DIFFUSE);
device->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
device->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);

// Then use the color from the previous texture, and the alpha from the mask.
device->SetTextureStageState(1, D3DTSS_COLOROP, D3DTOP_SELECTARG1);
device->SetTextureStageState(1, D3DTSS_COLORARG1, D3DTA_CURRENT);
device->SetTextureStageState(1, D3DTSS_COLORARG2, D3DTA_CURRENT);
device->SetTextureStageState(1, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
device->SetTextureStageState(1, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
device->SetTextureStageState(1, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);

This is possible in fixed function texture stages; sample the color texture in the first stage, and in the second stage, get the alpha from the second texture but take the color from the previous stage (D3DTA_CURRENT). The texture stage setup runs on the GPU hardware, unlike your proposed approach.

That sounds interesting, but I have no idea how to do something like that. Does this have to be done every frame before rendering or something? If you have any articles or anything on it or could post some pseudo code that would be very appreciated I'll try to do a bit of research on my own though.

A pixel shader would be more simple here, though

I know it would be, but the engine I'm using has no support for shaders whatsoever and it would be rather difficult to add it, especially with the time constraints on the project.

EDIT: Well I looked into it and I sort of understand what to do, but I can't figure out the mixture of texture stage states to get it working. This is what I have currently, which is resulting in no change:

device->SetTexture(0, fTexture);
device->SetTexture(1, fMaskTexture);

// Only use the color from the first texture.
device->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_MODULATE);
device->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_COLORARG2, D3DTA_DIFFUSE);
device->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
device->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
device->SetTextureStageState(0, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);

// Then use the color from the previous texture, and the alpha from the mask.
device->SetTextureStageState(1, D3DTSS_COLOROP, D3DTOP_MODULATE);
device->SetTextureStageState(1, D3DTSS_COLORARG1, D3DTA_CURRENT);
device->SetTextureStageState(1, D3DTSS_COLORARG2, D3DTA_DIFFUSE);
device->SetTextureStageState(1, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
device->SetTextureStageState(1, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
device->SetTextureStageState(1, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);

This is possible in fixed function texture stages; sample the color texture in the first stage, and in the second stage, get the alpha from the second texture but take the color from the previous stage (D3DTA_CURRENT). The texture stage setup runs on the GPU hardware, unlike your proposed approach.

That sounds interesting, but I have no idea how to do something like that. Does this have to be done every frame before rendering or something? If you have any articles or anything on it or could post some pseudo code that would be very appreciated I'll try to do a bit of research on my own though.

A pixel shader would be more simple here, though

I know it would be, but the engine I'm using has no support for shaders whatsoever and it would be rather difficult to add it, especially with the time constraints on the project.