Sign in to follow this  
helix

HLSL question -- passing data between vertex and pixel shaders

Recommended Posts

helix    301
I wrote a shader program in HLSL using DX effects that would blend various textures based on their Y position. As far as I can tell, I'm pretty sure my code is correct (hours spent going over my not-too-complex algorithm!). I am debugging it in Visual Studio .NET using effect edit and I see very strange results. This is my first unique shader program so I'm most likely just doing something wrong. So in my vertex shader, I'm declaring it as follows: void RenderTerrainVS( float4 Pos : POSITION, float3 Normal : NORMAL, float2 Tex : TEXCOORD0, out float4 oPos : POSITION, out float4 Diffuse : COLOR, out float2 oTex : TEXCOORD0, out float3 oNormal : TEXCOORD1, out float3 oViewPos : TEXCOORD2, out float3 oBlendInfo : TEXCOORD3 ) This is copied from an existing shader that basically just lights the scene. My addition is the last out variable, oBlendInfo. I just used the TEXCOORD3 register for the lack of any clue what I should use (and I saw some other shader in some microsoft sample using it and my shader didn't appear to be using it). Is this a reasonable one to use? How do I know how much data can fit in a register? How do I know which ones are available for use and won't clobber data stored for later? When I set the data in oBlendInfo, I do various calculations and end up with the following code: // Set the data to be passed on to the pixel shader oBlendInfo.x = (float)layer; // The layer we are in oBlendInfo.y = fBlendAmt; oBlendInfo.z = fLayerSection; layer is an int that I'm piggy backing on this variable mostly since I don't know what other registers to use for outgoing. So my pixel shader looks like this: void RenderTerrainPS( float4 Diffuse : COLOR, float2 Tex : TEXCOORD0, float3 Normal : TEXCOORD1, float3 Pos : TEXCOORD2, float3 BlendInfo : TEXCOORD3, out float4 oCol : COLOR0, out float4 oNormal : COLOR1, out float4 oPos : COLOR2 ) When BlendInfo comes in here, the members are all wacky. Let's say I pass (in the v shader) x = 3, y = 0.345, z = -10.0. When it comes into the pixel shader, my data now looks like x = 2.99843, y = 0.3411, z = -9.97949. What's up with that? As I step through the function, BlendInfo loses its data somehow (which shouldn't really matter since I already set variables with the data but it concerns me that it's disappearing). When looking at it in the debugger, after I set a variable with BlendInfo.x for example, the .x member is now a *. My guess is that it is an optimization because I'm not using .x anymore so it just now lives in the variable. Actually, I just had a thought, is it interpolating the values based on the pixel's location from the three vertices because I'm using a TEXCOORD register? I am loving HLSL and shaders but there are few things that just confuse the hell out of me!

Share this post


Link to post
Share on other sites
xyzzy00    307
Each register can store a float4. Any of the texcoord registers are available for use. Just make sure that your vertex shader outputs match up with your pixel shader inputs.

Yep, the value will get interpolated across the primitive. If you want it is constant across the primitive, make sure it is the same at all three vertices of the primitive.

Glad you like HLSL :)

xyzzy

Share this post


Link to post
Share on other sites
helix    301
Thanks for the reply. But I was hoping I was doing something wrong!! I can't figure out this bug in my shader program. Here's the source for it, maybe some of you experts can figure it out. I've gone through a number of iterations but this seems to be pretty close but not quite right...

Here's a screen shot of the problem. It's supposed to be blending seamlessly between each layer but you can see the stripping that is going on at the border.


Here is a snippet of my shader source.

const half g_fNumTerrainLayers = 4.0f; // How many textures are in the texture layers
//float g_fTerrainBasePoint = 0.0f; // Where the root of the terrain is (the Y coord of the origin of the root point)
float g_fLayerWidth = 1.0f / 4.0f; // How wide each texture layer is
float g_fBlendSection = 0.25f; // Blend on the upper and lower quarter of the texture layer

// First layer -- water
texture g_texLayer1; //< string name = "Media\\Textures\\water.tga"; >;

sampler2D texLayer1Sampler = sampler_state
{
Texture = <g_texLayer1>;
MipFilter = LINEAR;
MinFilter = LINEAR;
MagFilter = LINEAR;
};

// Second layer -- sand
texture g_texLayer2; //< string name = "Media\\Textures\\sand.tga"; >;

sampler2D texLayer2Sampler = sampler_state
{
Texture = <g_texLayer2>;
MipFilter = LINEAR;
MinFilter = LINEAR;
MagFilter = LINEAR;
};

// Third layer -- grass
texture g_texLayer3; //< string name = "Media\\Textures\\grass2.tga"; >;

sampler2D texLayer3Sampler = sampler_state
{
Texture = <g_texLayer3>;
MipFilter = LINEAR;
MinFilter = LINEAR;
MagFilter = LINEAR;
};

// Fourth layer -- rock
texture g_texLayer4; //< string name = "Media\\Textures\\mossy_rock.tga"; >;

sampler2D texLayer4Sampler = sampler_state
{
Texture = <g_texLayer4>;
MipFilter = LINEAR;
MinFilter = LINEAR;
MagFilter = LINEAR;
};

//
// Function to find the textures to use and the blend amount
// There are four layers (water, sand, grass, rock).
//
float4 TerrainTextureLayer( float3 fBlendInfo, float2 Tex : TEXCOORD0 ) : COLOR
{
float4 current = 0.0f; // The current layer's texture
float4 other = 0.0f; // The other layer's texture
float4 retVal = 0.0f; // The resulting texture color
int nLayer = (int)(fBlendInfo.x); // The layer we reside in
float fCurrBlendFactor = 1.0f - fBlendInfo.y; // Get the relative blend factor for the current layer.
float4 blendVals = float4( fCurrBlendFactor, fCurrBlendFactor, fCurrBlendFactor, fCurrBlendFactor );
float fLayerSection = fBlendInfo.z;

// Figure out which layer we reside in and blend between the result

// Layer 4 -- Mossy Rock
if( nLayer >= 3 )
{
// Add the Layer 4 texture
current = tex2D( texLayer4Sampler, Tex );

// No upper section
if( fLayerSection < 0.0f )
{
// Now blend the Layer 3 texture
other = tex2D( texLayer3Sampler, Tex );

// Now linearly interpolate between the textures
// This linearly interpolates between x and y, such that the return value is x when s is 0, and y when s is 1.
// lerp( x, y, s ) Returns x + s(y - x).
retVal = lerp( current, other, blendVals );
}
else
retVal = current;
}
else if( nLayer >= 2 ) // Layer 3 -- Grass
{
// Add the Layer 3 texture
current = tex2D( texLayer3Sampler, Tex );

// Upper section
if( fLayerSection > 0.0f )
{
// Layer 4 texture
other = tex2D( texLayer4Sampler, Tex );

// Now linearly interpolate between the textures
retVal = lerp( current, other, blendVals );
}
else if( fLayerSection < 0.0f ) // Lower section
{
// Layer 2 texture
other = tex2D( texLayer2Sampler, Tex );

// Now linearlly interpolate between the textures
retVal = lerp( current, other, blendVals );
}
else
retVal = current;
}
else if( nLayer >= 1 ) // Layer 2 -- Sand
{
// Add the Layer 2 texture
current = tex2D( texLayer2Sampler, Tex );

// Upper section
if( fLayerSection > 0.0f )
{
// Layer 3 texture
other = tex2D( texLayer3Sampler, Tex );

// Now linearly interpolate between the textures
retVal = lerp( current, other, blendVals );
}
else if( fLayerSection < 0.0f ) // Lower section
{
// Layer 1 texture
other = tex2D( texLayer1Sampler, Tex );

// Now linearly interpolate between the textures
retVal = lerp( current, other, blendVals );
}
else
retVal = current;
}
else // Layer 1 -- Water
{
// Add the texture but darken it as it gets deeper (it's water after all...)
// Negative clamps to 0.0f
current = tex2D( texLayer1Sampler, Tex ) * fBlendInfo.y; // Darken as it goes down (to show depth)

// Upper section (no lower section)
if( fLayerSection > 0.0f )
{
// Layer 2 texture
other = tex2D( texLayer2Sampler, Tex );

// Now linearly interpolate between the textures
retVal = lerp( current, other, blendVals );
}
else
retVal = current;
}

return retVal;
}

//--------------------------------------------------------------------------------------
// Shaders
//--------------------------------------------------------------------------------------

// Terrain

void RenderTerrainVS( float4 Pos : POSITION,
float3 Normal : NORMAL,
float2 Tex : TEXCOORD0,
out float4 oPos : POSITION,
out float4 Diffuse : COLOR,
out float2 oTex : TEXCOORD0,
out float3 oNormal : TEXCOORD1,
out float3 oViewPos : TEXCOORD2,
out float3 oBlendInfo : TEXCOORD3 )
{
// Output screen space position
oViewPos = mul( Pos, g_mWorldView );
oPos = mul( float4( oViewPos, 1.0f ), g_mProj );

// Transform normal to view space
oNormal = mul( Normal, (float3x3)g_mWorldView );

// Compute diffuse
Diffuse = 1.0f;//dot( g_vLightDirView, oNormal ); //Note: uncomment this to use lighting info

// Propagate tex coord
oTex = Tex;

// Get the layer and the blend factor of this layer's texture (from the top)
int layer = (int)(Pos.y / g_fLayerWidth);
// oBlendInfo.y = max( 1.0f, min( 0.0f, (Pos.y - ((float)layer * g_fLayerWidth)) / g_fLayerWidth ) ); // The blend value relative to the lower layer

// Find where in the texture layer we are
float fLayerAmt = Pos.y - ((float)layer * g_fLayerWidth);
fLayerAmt /= g_fLayerWidth; // Converts to 0.0 - 1.0 range

// Get the blend amount of our layer's texture
float fLayerSection = 0.0f; // Default in middle
float fBlendAmt = 1.0f;
if( fLayerAmt < g_fBlendSection )
{
fBlendAmt = fLayerAmt / (g_fBlendSection * 2.0f) + 0.5f;
fLayerSection = -10.0f; // Lower section
}
else if( fLayerAmt > g_fBlendSection * 3.0f )
{
fBlendAmt = (fLayerAmt - (g_fBlendSection * 2.0f)) / (g_fBlendSection * 2.0f) - 0.5f;
fLayerSection = 10.0f; // Upper section
}

// Set the data to be passed on to the pixel shader
oBlendInfo.x = (float)layer; // The layer we are in
oBlendInfo.y = fBlendAmt;
oBlendInfo.z = fLayerSection;
}


void RenderTerrainPS( float4 Diffuse : COLOR,
float2 Tex : TEXCOORD0,
float3 Normal : TEXCOORD1,
float3 Pos : TEXCOORD2,
float3 BlendInfo : TEXCOORD3,
out float4 oCol : COLOR0,
out float4 oNormal : COLOR1,
out float4 oPos : COLOR2 )
{
//
// Output pixel color
//
oCol = TerrainTextureLayer( BlendInfo, Tex ) * Diffuse;

//
// Output normal
//
oNormal = float4( normalize( Normal ), 1.0f );

//
// Output view position
//
oPos = float4( Pos, 1.0f );
}

Share this post


Link to post
Share on other sites
Jim Adams    440
I haven't looked at your code, but just so you know, the Microsoft HLSL compiler DOES have some bugs in it. If you're sure the code works, try rearranging the code and see if that helps.

Share this post


Link to post
Share on other sites
Jim Adams    440
To better iterate this - Microsoft's HLSL compiler is known to cause problems with certain sequences of commands. If you're having problems, and you know it should work, try reordering your code to see if it helps.

Jim

Share this post


Link to post
Share on other sites
JonStelly    127
You should ask this on the DX beta newsgroups, just to make sure it's not a beta bug. I'm assuming that's what you're using from the UI components, ignore me if I'm wrong.

Share this post


Link to post
Share on other sites
Jim Adams    440
They already know about it (I'm on the beta team). I have some code that won't compile correctly on any of the HLSL compilers to date (a, b, c, and rc0).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this