Hi!

I'm having problems getting shaders to work in my direct3d based engine. I used to use the fixed function pipeline but decided it was time to get into shaders, so I'm trying to get the simplest possible shaders to work in my engine.

I've looked at some tutorials (http://www.two-kings...graphics18.html for example) and it seems it shouldn't be too hard to get the easiest thinkable shader going.

When i try to render using my shaders, i get nothing, just a pitch black screen. I assume I'm missing something since I'm new to this, but I just can't figure it out since I've looked at multiple tutorials and examples and it seems that what I'm doing should work.

Here's some code:

Vertex shader:

struct VS_INPUT
{
float4 vPosition : POSITION;
float3 vNormal : NORMAL;
};
struct VS_OUTPUT
{
float4 vPosition : POSITION;
float4 vDiffuse : COLOR0;
};
// Global variables
float4x4 WorldViewProj;
float4 vLight;
VS_OUTPUT vs_main(in VS_INPUT In)
{
VS_OUTPUT Out;
// Output stuff
Out.vPosition = mul(In.vPosition, WorldViewProj); //apply vertex transformation
Out.vDiffuse = dot(vLight,In.vNormal);
return Out;
}



Pixel shader:


// Pixel shader input structure
struct PS_INPUT
{
float4 vPosition : POSITION;
float4 vDiffuse : COLOR;
};


// Pixel shader output structure
struct PS_OUTPUT
{
float4 Color[4] : COLOR0;
float Depth : DEPTH;
};


// Global variables
//texture2D Tex0;


// Name: Simple Pixel Shader
// Type: Pixel shader
// Desc: Fetch texture and blend with constant color
//
PS_OUTPUT ps_main(void)
{
PS_OUTPUT Out;

// Shader statements

// Write up to four pixel shader output colors
Out.Color[0] = (1.0f, 1.0f, 1.0f, 1.0f);
Out.Color[1] = (1.0f, 1.0f, 1.0f, 1.0f);
Out.Color[2] = (1.0f, 1.0f, 1.0f, 1.0f);
Out.Color[3] = (1.0f, 1.0f, 1.0f, 1.0f);

// Write pixel depth
Out.Depth = 1.0f;

return Out;
}


Shader initialization:


HRESULT gfx_Manager::InitShaders()
{
LPD3DXBUFFER m_ShaderBuffer;
// Create the vertex declaration
const D3DVERTEXELEMENT9 dec[] =
{
{0, 0, D3DDECLTYPE_FLOAT3, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_POSITION,0},
{0, 12, D3DDECLTYPE_FLOAT3, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_BLENDWEIGHT, 0},
{0, 24, D3DDECLTYPE_UBYTE4, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_BLENDINDICES, 0},
{0, 28, D3DDECLTYPE_FLOAT3, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_NORMAL,0},
D3DDECL_END()
};
g_pd3dDevice->CreateVertexDeclaration(dec, &m_vertexDeclaration);
// Create the vertex shader
if(FAILED(D3DXCompileShaderFromFile("LightedVS.hlsl", NULL, NULL, "vs_main",
"vs_2_0",
NULL, &m_ShaderBuffer, NULL, &constantTable)))
return E_FAIL;
if(FAILED(g_pd3dDevice->CreateVertexShader( ( DWORD* )m_ShaderBuffer->GetBufferPointer(), &g_pVertexShader)))
return E_FAIL;
m_ShaderBuffer->Release();

// Create the pixel shader
if(FAILED(D3DXCompileShaderFromFile("LightedPS.hlsl", NULL, NULL, "ps_main",
"ps_2_0",
NULL, &m_ShaderBuffer, NULL, NULL)))
return E_FAIL;
if(FAILED(g_pd3dDevice->CreatePixelShader( ( DWORD* )m_ShaderBuffer->GetBufferPointer(), &g_pPixelShader)))
return E_FAIL;
m_ShaderBuffer->Release();
return S_OK;
}


Relevant part of render function:

// Set the world matrix
g_pd3dDevice->SetTransform(D3DTS_WORLD, (*MatrixItr));
//communicate with shader (NEW)
D3DXMATRIXA16 matWorld, matView, matProj;
g_pd3dDevice->GetTransform(D3DTS_WORLD, &matWorld);
g_pd3dDevice->GetTransform(D3DTS_VIEW, &matView);
g_pd3dDevice->GetTransform(D3DTS_PROJECTION, &matProj);
D3DXMATRIXA16 matWorldViewProj = matWorld * matView * matProj;
constantTable->SetMatrix(g_pd3dDevice,
"WorldViewProj",
&matWorldViewProj);
float vLight[4] = {1,1,1,1};

constantTable->SetFloatArray(g_pd3dDevice, "vLight", vLight, 4);
// Render mesh
g_pd3dDevice->SetMaterial( &DefaultMaterial );
g_pd3dDevice->SetTexture( 0, 0 );
g_pd3dDevice->SetStreamSource( 0, (*MeshItr)->VertexBuffer, 0, sizeof( CUSTOMVERTEXWEIGHTED ) );
//g_pd3dDevice->SetFVF( D3DFVF_CUSTOMVERTEXWEIGHTED );
if(FAILED(g_pd3dDevice->SetVertexDeclaration(this->m_vertexDeclaration)))
return;
if(FAILED(g_pd3dDevice->SetVertexShader(this->g_pVertexShader)))
return;
if(FAILED(g_pd3dDevice->SetPixelShader(this->g_pPixelShader)))
return;
g_pd3dDevice->SetIndices( (*MeshItr)->IndexBuffer );
if(FAILED(g_pd3dDevice->DrawIndexedPrimitive( D3DPT_TRIANGLELIST,0,0, (*MeshItr)->VertexCount,0, (*MeshItr)->Polygons)))
{
OutputDebugString("DrawIndexedPrimitive FAILED\n");
}


Any help would be appreciated!

This may seem like a stupid question, but are your transforms setup properly to view your geometry?
Try making sure your world matrix is an identity matrix, your view is a slight "lookat" offset of that, and your projection is a proper perspective transform.
It's typical for some people (myself included) who are learning something new to overlook the simplest of things.
Usually ends in a smile + facepalm combo.

This may seem like a stupid question, but are your transforms setup properly to view your geometry?
Try making sure your world matrix is an identity matrix, your view is a slight "lookat" offset of that, and your projection is a proper perspective transform.
It's typical for some people (myself included) who are learning something new to overlook the simplest of things.
Usually ends in a smile + facepalm combo.

They should be all right, since it worked with the fixed function pipeline. Shouldn't be any difference when using shaders, right?

Try this:



float4 ps_main(void) : COLOR
{
return float4(1, 0, 0, 1);
}

Do you see anything red?

Tried it, still nothing :/

Any other ideas?

Thanks for your replies.

Does the debug runtime produce any error messages?

Can you post the code that sets up the view and projection transforms?

Have you tried debugging the shader with PIX?

Does the debug runtime produce any error messages?

Can you post the code that sets up the view and projection transforms?

Have you tried debugging the shader with PIX?

Debug runtime produces no error messages.

Matrix setup:


...
// Projection Matrix
D3DXMATRIXA16 matProj;
D3DXMatrixPerspectiveFovLH( &matProj, D3DX_PI / 4.0f, (float)SCREEN_WIDTH/SCREEN_HEIGHT, 1.0f, 1000.0f );
g_pd3dDevice->SetTransform( D3DTS_PROJECTION, &matProj );
...
// Setup the view matrix
D3DXMATRIXA16 matView;
D3DXMatrixLookAtLH( &matView, ActiveCamera->camPosition(), ActiveCamera->camOrientation(), ActiveCamera->camUp() ); // apparently this function is not good for this purpose. (but it works for now)
g_pd3dDevice->SetTransform( D3DTS_VIEW, &matView );
...


Projection matrix is setup at the beginning when initializing direct3d. The view matrix is setup in the render loop before drawing. As i said above this worked fine with the fixed function pipeline, shouldn't it then work with shaders too?

I'm new to shaders so I'm not that familiar with PIX, but i did play around a bit with it just now and discovered something weird. It claims there never was any draw calls. I tried capturing a single frame and capturing a range of frames and apparently there were no draw calls at all during any of those frames. I don't understand why this is since i call DrawIndexedPrimitive and it doesn't fail. :S

EDIT: Never mind, turns out i just don't know how to use PIX .

I think I've figured out what the problem is, but I'm not sure. I believe it's my vertex structure / vertex declaration that is the fault. It contains a bunch of blending data that the shader doesn't use. I tried using another vertex structure and it worked. So it seems I've sort of solved the problem, but i still need to get it to work with my own vertex structures, so if anyone knows where things went wrong in my first post I'd love to hear from you.

Check your messages.

Make sure your vertex declaration matches what's in the vertex data structure, and the correct side is used with SetStreamSource(). It looks like sizeof( CUSTOMVERTEXWEIGHTED ) should be 40 bytes.

If that doesn't help post the definition of CUSTOMVERTEXWEIGHTED here.