Jump to content
  • Advertisement
Sign in to follow this  
Kija

Vertex Shader Diffuse Color

This topic is 4840 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello, I've found a couple of threads covering almost the same problem but still can't find the solution. I'm really new with shaders so bare with me on this one. I took a tutorial on shaders from CodeSampler and implemented it in my own application. So the effect file specifies both a vertex and a pixel shader. Here's the (slightly modded) .fx file:
float4x4 worldViewProj : WorldViewProjection;
texture testTexture;

sampler Sampler = sampler_state
{
	Texture   = (testTexture);
	MipFilter = LINEAR;
	MinFilter = LINEAR;
	MagFilter = LINEAR;
};

struct VS_INPUT
{
	float4 position	: POSITION;
	float4 color	: COLOR0;
	float2 texture0 : TEXCOORD0;
};

struct VS_OUTPUT
{
	float4 position : POSITION;
	float4 color	 : COLOR0;
	float2 texture0  : TEXCOORD0;
};

struct PS_OUTPUT
{
	float4 color : COLOR;
};

VS_OUTPUT myvs( VS_INPUT IN )
{
	VS_OUTPUT OUT;

	OUT.position = mul( worldViewProj, IN.position );

	OUT.color = float4( 1.0, 0.0, 0.0, 0.5 );

	OUT.texture0 = IN.texture0;

	return OUT;

PS_OUTPUT myps( VS_OUTPUT IN )
{
	PS_OUTPUT OUT;

//	OUT.color = tex2D( Sampler, IN.texture0 ) * IN.color; // We don't want the texture atm
	OUT.color = IN.color; // This line makes the primitives white, why?
//	OUT.color = float4( 1.0, 0.0, 0.0, 0.5 ); // This line makes the primitives red with alpha = 50%

	return OUT;
}

technique Technique0
{
    pass Pass0
    {
		Lighting = FALSE;

		Sampler[0] = (Sample
		VertexShader = compile vs_1_1 myvs();
		PixelShader  = compile ps_1_1 myps();
    }
}


My primitives (two triangles in a strip) is rendered to screen space coordinates and thereby defined by this FVF and struct:
#define D3DFVF_GUIFVF (D3DFVF_XYZRHW | D3DFVF_DIFFUSE | D3DFVF_TEX1)

struct GuiVertex
{
	float x, y, z, rhw;
	DWORD color;
	float tu, tv;
};


And as you can see if you look carefully in the effect file, I change the OUT.color to red with 50% alpha. But if I (in the pixel shader) define the OUT.color as IN.color (which should be the output from the vertex shader, right?) the result is white with 100% alpha (opaque). But if I set the OUT.color (in the pixel shader) to red with 50% alpha that is also the result in my application. So why isn't the vertex shader outputting the red 50% alpha color to the pixel shader? In some other post someone had this problem but it was because the OUT.position of the vertex shader was wrong. He defined as OUT.position = IN.position instead of multiplying it with the world_view_projection matrix. I set the effect matrix from the application with this code:
	D3DXMATRIX mWorldViewProj, mWorld, mView, mProj;

	d3dDevice->GetTransform(D3DTS_PROJECTION, &mProj);
	d3dDevice->GetTransform(D3DTS_VIEW, &mView);
	d3dDevice->GetTransform(D3DTS_WORLD, &mWorld);

	mWorldViewProj = mWorld * mView * mProj;
	window_effect->SetTechnique("Technique0");
	window_effect->SetMatrix("worldViewProj", &mWorldViewProj);


Share this post


Link to post
Share on other sites
Advertisement
Hi there Kija,
How are you doing?

[The Problem]
The shader isn't doing what you want meaning that it's not rendering.

[The Solution]
The only problems that strike me as out of place are the following,

1) You are multiplying your already transformed coordinates with the WorldViewProjection matrix. You are telling direct3d that your coordinates are already transformed. So you don't need to multiply it.

OUT.position = mul( worldViewProj, IN.position );
Should be
OUT.position = IN.position;

2) When you have a pixel shader your colors are normally formed in there. So what you want to do is remove from your vertex declaration. Leave it in the pixel declaration.
float4 color : COLOR0;

3) In your pixel shader you just replace
OUT.color = IN.color;
with
//Remember it's in the form
//(A, R, G, B)
OUT.color = float4(1.0, 1.0, 0.0, 0.0);

PS:

float4x4 worldViewProj : WorldViewProjection;


struct VS_INPUT
{
float4 position : POSITION;
};

struct VS_OUTPUT
{
float4 position : POSITION;
};

struct PS_OUTPUT
{
float4 color : COLOR;
};

VS_OUTPUT myvs( VS_INPUT IN )
{
VS_OUTPUT OUT;

OUT.position = IN.position;

return OUT;

PS_OUTPUT myps( VS_OUTPUT IN )
{
PS_OUTPUT OUT;
OUT.color = float4(1.0f, 1.0f, 0.0f, 0.0f); //for 100% red.
return OUT;
}

technique Technique0
{
pass Pass0
{
Lighting = FALSE;

VertexShader = compile vs_1_1 myvs();
PixelShader = compile ps_1_1 myps();
}
}




I really hope this helps buddy
Good job so far.

Share this post


Link to post
Share on other sites
Quote:
Original post by Armadon

3) In your pixel shader you just replace
OUT.color = IN.color;
with
//Remember it's in the form
//(A, R, G, B)
OUT.color = float4(1.0, 1.0, 0.0, 0.0);


Actually, it's RGBA in HLSL, so Kija's right. Can't help for the main problem though.

Share this post


Link to post
Share on other sites
Hi there guys,
I do apologize, I just woke up when I made that reply. I was thinking about another constructor. I do apologize once again.

Your problem may still lie with the passing of your color from your vertex shader to your pixel shader.


hehehe, well done bleyblue2

Share this post


Link to post
Share on other sites
Big thanks for the replies guys. I agree that:

OUT.position = mul( worldViewProj, IN.position );

should be:

OUT.position = IN.position;

since the position is already transformed, just like Armadon said.

Even if I change this the result is still white, not red with alpha 50%. But if I change it so that the pixel shader sets the color, it works. But what I can't understand is why I can't export it from the vertex shader. Shouldn't it be possible? What if I set my pixel shader to NULL like this:

VertexShader = compile vs_1_1 myvs();
PixelShader = NULL;

I still get the same result.

Well my thoughts are in a slightly different path than yours. Since Armadon found that it was wrong to to multiply the IN.position with the matrix in the vertex shader, why did I still see the triangles? Now I've tried to set this in the vertex shader:

OUT.position = float4( 0.0, 0.0, 0.0, 0.0 );

And I still see the triangles, shouldn't this affect the vertices transformation? I have to be able to modify the vertex position / color in the vertex shader?

Well, as you know I'm new to this so I could be way wrong on this but still. I can't affect my triangles in _any_ way in the vertex shader part. Not position, not color, not texture coordinates.

Share this post


Link to post
Share on other sites
I'm not sure about this, but try applying the Lighting (TRUE) in the shader.

Oh.. and try to remove the RHW (D3DFVF_XYZ).. you really don't need to set the FVF with shaders (it's for Fixed T&L), works, at least with asm shaders. :)

Share this post


Link to post
Share on other sites
Hello, it didn't matter how I set the lighting in the effect file but that could be because I have lighting disabled in d3d? I guess that is FFP and should be controlled only by shaders. However I'm just making the switch to shaders so I have some ffp left. For example lighting = true / false and transformations.

And about the other part, removing the RHW and that FVF isn't really necessary with shaders. I don't get it. I want to specify my vertex data in the direct3d code and by that I need the vertex struct to handle all data necessary. In my case that has to be x, y, z, rhw, color, tu and tv? And I still need to use d3dDevice->SetFVF(...). So if I remove the FVF, can I just skip that call? And if I remove the rhw from the struct and just have x y z as the position. The shader would take in a float3 position and then set the fourth (rhw) component to 1.0f (for screen space coords)?

Oh my... I'm getting even more confused by this :)

I guess the problem still is, that the vertex shader isn't working... But the "best" evidence of this has to be that if I set the OUT.position in the vertex shader to 0, 0, 0, 0 I still see it in the application. Should it be that way if the vertex shader is working correctly?

Share this post


Link to post
Share on other sites
Still asuming that the HLSL doesn't need the FVF (haven't used HLSL), anyway it seems like you declare the vertex format at the same time (INPUT) in th shader. Just set float3 value for the position.

Share this post


Link to post
Share on other sites
I think your problem is because of the D3DFVF_RHW flag. This flags barely means that your vertices are transformed by your application. But because they are already transformed, they won't pass through the vertex shader ! Removing this flag, you can simulate the FFP by using Armadon's simple VS, i.e. :

VS_OUTPUT myvs( VS_INPUT IN )
{
VS_OUTPUT OUT;

OUT.position = IN.position;

return OUT;
}

So, in conclusion :
- don't use the RHW flag with vertex shaders, even if your vertices are already pre-transformed (use only D3DFVF_XYZ), else the transformation pipeline, programmable or FF, will be bypassed;

- the vertex shader for pre-transformed vertices should output the same position as provided in the VS input.

I think you can now output your color from the vertex shader.

EDIT: oh, and btw Arex, HLSL doesn't need no vertex declaration itself, but using any draw*** call requires you to set one - by using SetVertexDeclaration or SetFVF (who calls in turn SetVertexDeclaration nowadays).

Share this post


Link to post
Share on other sites
After reading this thread, to me it sounds MOST like you're not setting the shader effects. Could we see the code where you:

-Initialize the ID3DXEffect
-Set the technique
-Set the pass and
-Draw the object

please?

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!