Once you start with shaders, you have to leave behind all of the fixed-function stuff. This includes texture stage states. If you want to multiply your diffuse lighting by a vertex color, then you need to pass the vertex color to your pixel shader and multiply your lighting by it.


sampler DiffuseSampler;
void main(in float2 vTex : TEXCOORD0, in float4 vColor : COLOR0, out float4 oCol : COLOR0)
{
oCol=tex2D(DiffuseSampler, vTex);
float aGrey=(oCol.r+oCol.g+oCol.b)/3;
oColor.rgb = vColor.rgb * aGrey;
}


Untested, but you should be able to simplify while also correcting for luminance with something like this:



sampler DiffuseSampler;

void main(in float2 vTex : TEXCOORD0,in float4 vColor : COLOR0,out float4 oCol : COLOR0)
{
float3 luminance = float3(0.30, 0.59, 0.11);

tColor = tex2D(DiffuseSampler, vTex);

tColor = float4(float3(dot3(luminance, tColor)), tColor.a);
vColor = float4(float3(dot3(luminance, vColor)), vColor.a);

oCol = tColor * vColor;
}

My syntax might be off, but the algorithm should be sound. If you don't want gamma correction, you can do "float3 luminance = float3(1, 1, 1) / 3;" instead.

dot3 multiplies component-wise, then sums the components, so you can compute the average of 3 components using (1/3, 1/3, 1/3) in one step, or use the weighted distribution like I did to correct luminance at the same time.

Using the above, you'd probably have to change things a fair bit in order to fade from full color to grayscale. look up the "lerp" function.

I can't speak to performance of this with any authority, but I think the HLSL and driver-level JIT compiler is pretty good at eliminating overhead due to vector-size transforms.

With shaders you can use variables called "shader constants". Their value can be set by your C++ program, but are constant during the execution of a shader. To use it, you just declare a float in your shader:

sampler DiffuseSampler;

float mSaturation;

void main(in float2 vTex : TEXCOORD0, in float4 vColor : COLOR0, out float4 oCol : COLOR0)
{
oCol = tex2D(DiffuseSampler, vTex) * vColor;
oCol.rgb = lerp(dot(oCol.rgb, 0.333f), oCol.rgb, mSaturation);
}


Now the way this works, is that the device has a set of constant registers that can be set from the app using SetPixelShaderConstantF (or SetVertexShaderConstantF for vertex shaders). Each of these registers is a float4, so when you call SetPixelShaderConstantF you set at least 4 floats at a time. When you declare a non-static global variable in your shader, that variable gets mapped to a register by the compiler. Typically it will just assign variables to sequential registers starting with 0, but you can also manually assign variables to registers with this syntax:


float mSaturation : register(c0);


To set this variable, you would call SetPIxelShaderConstantF and pass 0 as the StartRegister parameter. If you want, you can also query the register for a variable at runtime using ID3DXConstantTable.

Texture sampler also work in a similar way, in that they will be mapped to sampler registers s0 through s15. This register then corresponds to the index you pass to SetTexture.

The way that pixel shader input parameters work, is that they are mapped to the outputs of your vertex shader. For a vertex shader, the input parameters are mapped to elements of vertices in the vertex buffer. This mapping is done using semantics, So take this simple vertex shader:

float4x4 mWorldViewProjection;

void vsmain(in float3 vPosition : POSITION, in float2 vTex : TEXCOORD0, in float4 vColor : COLOR0,
out float4 oPosition : POSITION, out float2 oTex : TEXCOORD0, out float4 oColor : COLOR0)
{
oPosition = mul(float4(vPosition, 1.0f), mWorldViewProjection);
oTex = vTex;
oColor = vColor;
}


The input parameters will be taken from the vertex buffer, and will have the value of the element that was declared with the POSITION usage in the vertex declaration (or it will be based on the FVF code, if you're using those instead of vertex declarations). The vertex shader transforms the vertex position to clip space and outputs it, which ultimately determines where the vertex ends up in screen space and consequently how the triangle gets rasterized. The texture coordinate and diffuse color are simply passed through. Parameters that get passed through are then accessible in the pixel shader, as long as the pixel shader declares an input with a matching semantic. So if we were to use your pixel shader, the vTex and vColor parameters would get mapped to the vertex texture coordinate and diffuse color that got passed through in the vertex shader. The actual value received by the pixel shader will be interpolated based on where the pixel lies on the triangle.