According to the DX9SDK regarding D3DTOP_DOTPRODUCT3 and bumpmapping: "in DirectX 8.1 shaders you can specify that the output be routed to the .rgb or the .a components or both (the default)." I would like the output to be routed to only the .rgb, and not the .a component. How do I change this?

Just popping this one back to the top to see if anybody else has an idea on how to change this.

What I am trying to do is a two stage image:

Stage 0 is a DOT3PRODUCT bumpmap. Since its output is automatically being written to the alpha channel, it is affecting Stage 1, which is the normal, colored picture. Only Stage 1 is coming out translucent, because of the Stage 0 being written to alpha.

Anybody have an idea on how to fix this? Any input is greatly appreciated.

I know the docs used to say that it wrote to both color and alpha, and that alphaops on the same stage as a DOT3 were ignored. I'm not sure when it changed to it's current wording, or even if the current wording is accurate.

With a pixel shader, you can indeed write to .rgb, .rgba, or .a. With the fixed pipeline though, I'm pretty sure it always writes to all channels. The header file (press F12 with D3DTOP_DOTPRODUCT3 under the cursor) states it writes to all 4 channels, and can only be a colorop.

As for fixing your problem, what are you expecting in the alpha? Setting stage 0 to SELECT, CURRENT means the same thing as SELECT, DIFFUSE. If you are expecting your vertex/material alpha values, then just set stage 1's alphaop to select diffuse, or to modulate diffuse with a texture.

What you need depends on what you expected to be in alpha from stage 0, and what you are doing to alpha in stage 1.

If you don't want blending, you should just turn it off, as it's faster to render without blending, and the contents of the alpha can be safely ignored.

Thanks for the reply!

Here's the setup:

Stage 0, COLOROP, DOTPRODUCT3
Stage 0, COLORARG1, TEXTURE
Stage 0, COLORARG2, TFACTOR
Stage 0, ALPHAOP, MODULATE
Stage 0, ALPHAARG1, TEXTURE
Stage 0, ALPHAARG2, DIFFUSE

Stage 1, COLOROP, MODULATE
Stage 1, COLORARG1, TEXTURE
Stage 1, COLORARG2, DIFFUSE
Stage 1, ALPHAOP, MODULATE
Stage 1, ALPHAARG1, TEXTURE
Stage 1, ALPHAARG2, DIFFUSE

What I am trying to do is add bumpmapping to a 2D Sprite. So I need to keep alpha blending on to occlude the "black" area around the drawn sprite.... or am I misunderstanding the operation of alpha blending?

Alpha blending will allow you to cut out bits of your sprite, but also to partially blend with what's behind it. Alpha testing allows you to cut out bits of your sprite, with no blending. Alpha testing is often preferred when possible because you don't have to have perfect Z sorting... the Z buffer will take care if it. With blending, you must be very careful about the order you draw things. They can also be combined, blending all but the lowest alpha values, which you just have it skip via an alpha test.

To use alpha test, set these states:

D3DRS_ALPHATESTENABLE, true
D3DRS_ALPHAFUNC, D3DCMP_GREATER
D3DRS_ALPHAREF, 240

This says to keep any pixel whose alpha is over 240.

Are you using a colorkey when loading the image, to convert black pixels into alpha? If you are, remember the color key needs to be 0xFF000000, not 0x00000000, ie: alpha must be 255, not 0. A colorkey setting of 0 means "don't color key". By settings the alpha you're saying "make my black pixels transparent".

As for your texture stages, they should be okay. It will even allow you to fade out sprites based on material or vertex color alpha values. If you just want your texture's alpha, change stage 1 to SELECTARG1, TEXTURE. You should also set stage 2 colorop and alphaop to disable, to clean up whatever may have been set previously.

Whether I use alpha testing, or alpha blending, the problem is still that the DOT3 results are written to .rgba, instead of just .rgb (which is what I am after).

Since the results of the DOT3 include an alpha of 0 - 255, that means if I use alpha testing, any part of my sprite that has an alpha less than my cutoff point(alpha < 240 in your example) will not be drawn at all, so it leaves me with large portions of my sprite simply undrawn. With alpha blending, those portions come through as translucent.

Even though the documentation says that you can turn off writing to .a, I can't find anywhere it says how do actually do this.

My current workaround is to draw a completely solid sprite first, and then draw the two stage DOT3 + sprite on top of that. It works, but it means I have to draw the same thing twice, slowing things down.

If I can't find a way to turn off writing to the .a component, do you have any other ideas how to reduce the sprite drawing to just one operation?

Thanks again!

Alpha blending and testing works with the result of the last texture stage. While the DOT3 writes to RGBA, you don't care. You just don't use that result in your next stage. Stage 1 is using just selecting the texture (or possibly modulating texture and diffuse) alpha, and ignoring what the DOT3 did on stage 0. The DOT3 will have absolutely no effect on alphatest/blend in this case.

If your alpha is in your normal map, and not your diffuse map, you'll need to use 3 stages. One to DOT3, one to select the alpha, and a third to apply your color. If your alpha is in your diffuse map, then eveything's fine with 2 stages.

If you're having trouble, it's more likely that you haven't set stage 1's TEXCOORDINDEX to 0 to tell D3D that stage 1 should use the first (and likely only) set of UVs in your vertex. By default it sets this to 1. If you don't have 2 sets of UVs, it will just sample your texture's upper left corner, over and over.

Without more code, or screenshots, or something, all I can do is guess what's wrong, but you can safely ignore DOT3 being a problem.

It was a while ago, but check out this thread for how I got a solution to this very problem. Take a look at the last post to see the stages I set.
Hope it helps,
Steele.