I just started making the effect files for my game,the first one is gonna be the terrain texture blender,but it kinda confuses me.What exactly are the in and out keywords?I downloaded a sample blending effect that blends 2 textures according to a blendmap.My idea is to make it blend tile textures on a grid,according to a tilemap(each square on the grid being some kind of terrain type-grass,rock,dirt,sand).Here is what i got so far:
You declare variables at the top that you later link to the variables you want to operate on in your C++ code,you make a struct for vertex shader and for pixel shader with members corresponding to your custom vertex declaration in C++,you make functions that operate on the structs/variables/whatever and then you compile the shader.
I saw a main function in only a few of the sample .fx files I downloaded,so i guess it's not manditory.Does this mean that when you use an .fx file ALL functions in it are called?

The sample I've been looking at:


//=============================================================================
// Terrain.fx by Frank Luna (C) 2004 All Rights Reserved.
//
// Blends three textures together with a blend map.
//=============================================================================

uniform extern float4x4 gViewProj;
uniform extern float3 gDirToSunW;
uniform extern texture gTex0;
uniform extern texture gTex1;
uniform extern texture gTex2;
uniform extern texture gBlendMap;
static float gTexScale = 16.0f;
sampler Tex0S = sampler_state
{
Texture = <gTex0>;
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = POINT;
AddressU = WRAP;
AddressV = WRAP;
};
sampler Tex1S = sampler_state
{
Texture = <gTex1>;
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = POINT;
AddressU = WRAP;
AddressV = WRAP;
};
sampler Tex2S = sampler_state
{
Texture = <gTex2>;
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = POINT;
AddressU = WRAP;
AddressV = WRAP;
};
sampler BlendMapS = sampler_state
{
Texture = <gBlendMap>;
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = POINT;
AddressU = WRAP;
AddressV = WRAP;
};

struct OutputVS
{
float4 posH : POSITION0;
float2 tiledTexC : TEXCOORD0;
float2 nonTiledTexC : TEXCOORD1;
float shade : TEXCOORD2;
};
OutputVS TerrainVS(float3 posW : POSITION0, // We assume terrain geometry is specified
float3 normalW : NORMAL0, // directly in world space.
float2 tex0: TEXCOORD0)
{
// Zero out our output.
OutputVS outVS = (OutputVS)0;

// Just compute a grayscale diffuse and ambient lighting
// term--terrain has no specular reflectance. The color
// comes from the texture.
outVS.shade = saturate(max(0.0f, dot(normalW, gDirToSunW)) + 0.3f);

// Transform to homogeneous clip space.
outVS.posH = mul(float4(posW, 1.0f), gViewProj);

// Pass on texture coordinates to be interpolated in rasterization.
outVS.tiledTexC = tex0 * gTexScale; // Scale tex-coord to tile.
outVS.nonTiledTexC = tex0; // Blend map not tiled.

// Done--return the output.
return outVS;
}
float4 TerrainPS(float2 tiledTexC : TEXCOORD0,
float2 nonTiledTexC : TEXCOORD1,
float shade : TEXCOORD2) : COLOR
{
// Layer maps are tiled
float3 c0 = tex2D(Tex0S, tiledTexC).rgb;
float3 c1 = tex2D(Tex1S, tiledTexC).rgb;
float3 c2 = tex2D(Tex2S, tiledTexC).rgb;

// Blendmap is not tiled.
float3 B = tex2D(BlendMapS, nonTiledTexC).rgb;
// Find the inverse of all the blend weights so that we can
// scale the total color to the range [0, 1].
float totalInverse = 1.0f / (B.r + B.g + B.b);

// Scale the colors by each layer by its corresponding weight
// stored in the blendmap.
c0 *= B.r * totalInverse;
c1 *= B.g * totalInverse;
c2 *= B.b * totalInverse;

// Sum the colors and modulate with the shade to brighten/darken.
float3 final = (c0 + c1 + c2) * shade;

return float4(final, 1.0f);
}
technique TerrainTech
{
pass P0
{
vertexShader = compile vs_2_0 TerrainVS();
pixelShader = compile ps_2_0 TerrainPS();
}
}


The only two functions in an .FX file that are guaranteed to be called are the two functions that you list at the end (in "pass P0"), "vertexShader" and "pixelShader" in your case. You can, however, have any number of helper functions etc. above these and call them from either of the two 'main' functions (if you want to call them that).

It's important to keep in mind how the data flows through the shader - I personally use four structs in my HLSL shaders to keep the code readable, your example shader uses only two (the output structs, these are mandatory). For the input to both vertex and pixel shader it just lists the variables as function parameters. I replace that mess with two additional structs so in total I have "vertex_in", "vertex_out", "pixel_in" and "pixel_out", each a struct holding exactly the data that the vertex or pixel shader passes in or out.

This is how the two vertex shader structs would look like for your shader:

[source lang="cpp"]
struct InputVS
{
float3 posW : POSITION0, // we assume terrain geometry is specified
float3 normalW [font=courier new,courier,monospace]: NORMAL0, // directly in world space
float2 tex0 : TEXCOORD0[/font]
};

struct OutputVS
{
float4 posH : POSITION0;
float2 tiledTexC : TEXCOORD0;
float2 nonTiledTexC : TEXCOORD1;
float shade : TEXCOORD2;
};
[/source]

You don't have to compile the shader yourself before running your game, D3DX can do this for you via "D3DXCreateEffectFromFile". Make sure to read any error messages if that function (and the shader compile) fails, it'll tell you when there's an error in your shader and where it is!

This is how I load shaders in my games:

[source lang="cpp"]
// NOTE: remove the space wherever there should be two < right after another! I had to include these because otherwise the code wouldn't show up properly in my post. Sorry for that!

void Shader::CreateFromFile ( std::string filename )
{
ID3DXBuffer *error_buffer = NULL;

if ( FAILED ( D3DXCreateEffectFromFile ( window->GetDevice ( ), filename.c_str ( ), NULL, NULL, SHADER_COMPILE_FLAGS, NULL, &effect, &error_buffer ) ) )
{
std::stringstream s;

if ( !error_buffer )
s < < "Failed to load shader '" < < filename.c_str ( ) < < "'...";
else
{
s < < "Failed to compile shader '" < < filename.c_str ( ) < < "'..." < < std::endl < < std::endl;
s < < "The following errors were encountered during compilation:" < < std::endl < < std::endl;
s < < ( const char *) error_buffer->GetBufferPointer ( );
}

Error ( s.str ( ) );
window->Quit ( );
return;
}

if ( !effect )
{
std::stringstream s;
s < < "Failed to create effect from file '" < < filename.c_str ( ) < < "'...";
Error ( s.str ( ) );
window->Quit ( );
return;
}

if ( error_buffer )
error_buffer->Release ( );

delete error_buffer;
}
[/source]

It's practical this way because I can have the .FX file open in a text editor or my compiler IDE, make a change to the shader and run the .exe again, no need to re-compile or run the HLSL compiler or anything.

Apart from that, I don't quite get your texture blending plan quite yet. The easiest way to do this usually is have the blendmap be just different shades of gray where the brightness of a pixel determines how much of texture A or texture B is mixed in. Looks like the shader you posted should do exactly that!

That's about all I can think of, hope this helps!

in and out just specifies which direction data goes (useless keyword in my opinion).

Lets say you have vertex shader main:

void main(in float3 posA : POSITION0, in float3 normalA : NORMAL, out float4 posB : POSITION0, out float3 normalB : NORMAL) {
posA = posB;
normalA = normalB;
}

It just means vertex shader receives posA and normalA, and returns posB and normalB. In my opinion return value makes much less mess, than in/out keywords.


technique defines which shaders will be executed. In your provided example that would be:

technique TerrainTech {
pass P0 {
vertexShader = compile vs_2_0 TerrainVS();
pixelShader = compile ps_2_0 TerrainPS();
}
}

It tells effect compiler to compile shader TerrainVS and Vertex Shader 2.0 and use it as for TerrainTech technique, same goes for TerrainPS.

Yeah thanks,I think I got it.

Apart from that, I don't quite get your texture blending plan quite yet.

My plan is basically making a tilemap.For instance entire zone is 512x512 meters big.1 tile is 1 square meter.So the terrain is an array of structs:


struct Tile
{
float tileHeight;
unsigned char typeID;
unsigned char tileVariation;
};
Tile Terrain[512][512];


so then I can just use .NET to make a quick editor program to draw my game terrain using a square brush and place different tile types(grass,rock,etc.)

If you had a vertex or pixel shader that used the same structure type as the input and an output then you would use the "IN " and "OUT" keywords as a prefix to determine which var in the structure you were referring. I would assume this had to do with older versions having significant memory limitations and the IN and OUT functionality allowed for flexibility in this scope.

EDIT: wrong thread.