For pixel shaders you could use the HLSL function clip() - although might require ps2.0 for what you want to do (ie, for ps1.1 clip() uses texkill which wouldn't work because it'd need to modify texcoords in pixel shader which can't do in ps1.1).
Don't know about fixed function - you could have diffuse and specular in their own passes - add all the diffuse passes first, then multiply in the diffuse texture and then add in all specular passes. But you're probably trying to avoid those extra passes.

If you are using the type of specular which is a constant color added to the (diffuse light term * diffuse texture color), then it seems like you have a problem. You could do it with Multiple Render Targets, and create a special specular level render target, but not much hardware supports those.

On the other hand, maybe what you should do is use a light equation more like

(diffuse light term + spec light term) * light color * texture color

so you can compute the entire lighting term at once and don't need to use the alpha to save the specular and do it later. This may make more sense for some materials anyway, and it may be more efficient for your rendering system. I think if you try this you'll find it's a little cleaner.

SimmerD: I think his problem is that when drawing the lighting pass, if he is outputing an alpha for specular he cannot also output another alpha which is to be used for pixel alpha test. So he can't get the alpha test in the lighting pass; he could, as you say, get it in the diffuse color pass.

Hardguy: Good point, that equation won't saturate the color. But what you could do is divide the color value you write for lighting by 2 (or 4, or whatever is convenient), and then later, when you multiply by the diffuse texture, multiply by 2. This way you're effectively compressing and re-expanding the light value. This should be pretty easy to do in a pixel shader. Of course, this limits the amount of saturation you could do, but it shouldn't be a problem unless you have lots of dark textures which are lit very brightly (which won't look good anyway).

However the method you came up with, using the ztest, is good too, and you don't lose any color accuracy like you would with the "compression" method. There shouldn't be any problem with precision as long as you use the exact same matrices for each pass, and all the math works the same. I am rendering stuff with a Z-equal test myself at the moment actually :). And in fact, it seems like the performance should be better if anything, since the hardware can do the Z-test before the pixel shader is invoked but the alpha test must be done after.

Quote:Original post by Hardguy
Soiled: Never used HLSL so I have no idea what clip() does :) How does it helpme?

Use clip(x) in pixel shader as a replacement for the alpha-test - that way you can output specular to alpha like you currently do and it won't interfere with pixel rejection test.
clip(x) discards fragment if any component of vector (or scalar) x is less than zero so if you had alpha-test of Pass(alpha > 0.5) then use clip(alpha-0.5) instead. Though I think this only works on ps2.0 because the clip() HLSL function uses the texkill assembly instruction which requires 'x' to be a texture coord and in ps1.1 texture coords are read-only whereas in ps2.0 (actually ps1.4 and above) 'x' can be a temporary register which is read/write. You need to be able to write to 'x' because presumably 'x' comes from a texture or result of pixel shader calculation.