Hello

I'm having trouble with writing a basic vertex shader using D3D9 - I was hoping somebody could help me find the problem.

First of all here's the FX file I'm using:



//-----------------------------------------------------------------------------
// Effect File Variables
//-----------------------------------------------------------------------------

float4 LightDir = {0.5f, 0.5f, -0.5f,1.0f};
float4 LightColor = {0.7f, 0.7f, 0.7f, 1.0f};

float4x4 gWorldViewProj : WORLDVIEWPROJ;
float4x4 gWorldMatrix : WORLD;
float4x4 gViewMatrix : VIEW;
float4x4 gProjMatrix : PROJECTION;
float3 gViewPos : CAMERAPOSITION;

//-----------------------------------------------------------------------------
// Vertex Definitions
//-----------------------------------------------------------------------------

struct VS_INPUT
{
float3 vPos : POSITION;
};

struct VS_OUTPUT
{
float4 vPos : POSITION;
};

//-----------------------------------------------------------------------------
// Simple Vertex Shader
//-----------------------------------------------------------------------------

void myvs( float3 vPos : POSITION, out VS_OUTPUT OUT )
{
OUT.vPos = mul( gWorldViewProj, float4(vPos,1.0f) );
}

//-----------------------------------------------------------------------------
// Simple Pixel Shader
//-----------------------------------------------------------------------------

float4 myps( in VS_OUTPUT IN ) : COLOR
{
return LightColor;
}

//-----------------------------------------------------------------------------
// Simple Effect (1 technique with 1 pass)
//-----------------------------------------------------------------------------

technique lol
{
pass P0
{
VertexShader = compile vs_2_0 myvs();
PixelShader = compile ps_2_0 myps();

//ZEnable = false;
}
}


As you can see, it's nothing special. Before I show you the code for rendering with the shader, let me say that if I remove the vertex shader and only render pixel shader it works fine. So I can only assume that my matrices are incorrect.

Here's how I render with the shader:


if( !m_shader->SetTechnique( "lol" ) )
DEBUG_INFO( DEBUG_FATAL, "Failed to set effect technique" );

m_shader->SendRenderData( &matWorld );

unsigned int iPasses = 0;
if( !m_shader->Begin( &iPasses ) )
DEBUG_INFO( DEBUG_FATAL, "Failed to set effect technique" );

for( unsigned int i = 0; i < iPasses; i++ )
{
if( !m_shader->Pass( i ) )
DEBUG_INFO( DEBUG_FATAL, "Failed to start pass %i", i );

//m_device->SetTransform( D3DTS_WORLD, &matWorld );
m_device->SetVertexDeclaration( Application::Get()->GetRenderer()->GetVertexDeclaration() );
m_device->SetFVF( VOXEL_VERTEXFORMAT );
m_device->SetStreamSource(0, m_vertexBuffer, 0, sizeof(VoxelVertex));
m_device->SetIndices( m_indexBuffer );

HRESULT hr = m_device->DrawIndexedPrimitive(D3DPT_TRIANGLELIST, 0, 0, m_vertices.size(), 0, m_indices.size()/3 );
if( FAILED( hr ) )
DEBUG_INFO( DEBUG_FATAL, "Failed to draw chunk 0x%x", hr );

if( !m_shader->EndPass( ) )
DEBUG_INFO( DEBUG_FATAL, "Failed to end pass %i", i );
}

m_shader->End( );


No errors are returned. At any point. I just see nothing geometry on the screen.

Where I send my data to the shader:


void Shader::SendRenderData( D3DXMATRIX * world )
{
if( !m_application )
return;

if( !m_effect )
return;

Renderer * pDevice = m_application->GetRenderer( );

if( !pDevice )
return;

D3DXMATRIX matWorld = world ? *world : pDevice->GetWorldTransform( );
D3DXMATRIX matView = pDevice->GetViewTransform( );
D3DXMATRIX matProj = pDevice->GetProjectionTransform( );

D3DXMATRIX matWorldViewProj = matWorld * matView * matProj;

m_effect->SetMatrix( "gWorldMatrix", &matWorld );
m_effect->SetMatrix( "gViewMatrix", &matView );
m_effect->SetMatrix( "gProjMatrix", &matProj );
m_effect->SetMatrix( "gWorldViewProj", &matWorldViewProj );
//m_effect->SetVector( "gViewPos", &pDevice->GetViewVector( ) );

D3DXMATRIX mwvp;
m_effect->GetMatrix( "gWorldViewProj", &mwvp );

m_effect->CommitChanges( );
}


I'm not very experienced with shaders in particular, however I can't see any obvious problems.

I stepped through with a debugger and the final WVP matrix is valid and looks something like this:

_11 0.50682366 float
_12 -0.75278497 float
_13 -0.81174475 float
_14 -0.81171763 float
_21 0.00000000 float
_22 -1.5269465 float
_23 0.47204468 float
_24 0.47202891 float
_31 -1.1960893 float
_32 -0.31898057 float
_33 -0.34396380 float
_34 -0.34395233 float
_41 0.00000000 float
_42 1368.1440 float
_43 -423.05203 float
_44 -422.93790 float


I look forward to your responses.

Thanks

I'm not that familiar with the fixed function pipeline, but I think you shouldn't be setting a FVF format at all if you are using a vertex shader. Try removing that and see if it helps.

Outside of that, I would suggest taking a frame capture with PIX, then stepping through the draw call and see what the geometry looks like at each stage of transformation. That should give you some clues about what is going wrong.

I'm not that familiar with the fixed function pipeline, but I think you shouldn't be setting a FVF format at all if you are using a vertex shader. Try removing that and see if it helps.

Outside of that, I would suggest taking a frame capture with PIX, then stepping through the draw call and see what the geometry looks like at each stage of transformation. That should give you some clues about what is going wrong.

Unfortunately that didn't seem to do it. Thanks for the fast reply though.

If I can't resolve it then I guess I'll resort to using PIX.

It's been a while since I've done D3D9 effects, but I think that your transformation is the wrong way around. Try changing your vertex shader from
void myvs( float3 vPos : POSITION, out VS_OUTPUT OUT ) {
OUT.vPos = mul( gWorldViewProj, float4(vPos,1.0f) );
}
to
void myvs( float3 vPos : POSITION, out VS_OUTPUT OUT ) {
OUT.vPos = mul( float4(vPos,1.0f), gWorldViewProj );
}

I'm affraid that didn't work either. Thanks for the suggestion.

Try commenting out the line
m_effect->CommitChanges( );

when sending data to your shader. You only need that if you perform changes during a pass (multiple subsets etc.). Calling it incorrectly screwed up my render pretty badly a while ago..

Or change your input vertex format to float4:

struct VS_INPUT
{
float4 vPos : POSITION;
};

struct VS_OUTPUT
{
float4 vPos : POSITION;
};

//-----------------------------------------------------------------------------
// Simple Vertex Shader
//-----------------------------------------------------------------------------

void myvs( float4 vPos : POSITION, out VS_OUTPUT OUT )
{
OUT.vPos = mul( gWorldViewProj, vPos );
}

No dice. Thank you for the reply.

I guess I'll post some my shader class just incase I messed something up in there:


////////////////////////////////////////////////////////////////////////////////
// Shader::Shader
//! Constructor
//!
//! \return -
////////////////////////////////////////////////////////////////////////////////
Shader::Shader( )
{
m_effect = NULL;
m_device = NULL;
m_application = NULL;
m_fileSystem = NULL;
m_technique = NULL;

m_hasTechnique = false;
}

////////////////////////////////////////////////////////////////////////////////
// Shader::~Shader
//! Destructor
//!
//! \return -
////////////////////////////////////////////////////////////////////////////////
Shader::~Shader( )
{

}

////////////////////////////////////////////////////////////////////////////////
// Shader::Init
//! Initialize a shader
//!
//! \param * name -
//! \return void -
////////////////////////////////////////////////////////////////////////////////
void Shader::Init( char * name )
{
m_application = Application::Get( );

if( !m_application )
DEBUG_INFO( DEBUG_FATAL, "Could not get the application pointer" );

m_fileSystem = m_application->GetFileSystem( );

if( !m_fileSystem )
DEBUG_INFO( DEBUG_FATAL, "Could not get file system" );

m_device = (IDirect3DDevice9*)m_application->GetDeviceCreator( )->GetRenderDevice( )->GetDevice( );

if( !m_device )
DEBUG_INFO( DEBUG_FATAL, "Could not get the render device" );

strcpy_s( m_name, sizeof( m_name ), name );

if( !m_fileSystem->GetDirectoryFile( DATA_DIR_EFFECTS, name, m_absPath, sizeof( m_absPath ) ) )
DEBUG_INFO( DEBUG_FATAL, "Could not get directory file" );

LPD3DXBUFFER pBufferErrors = NULL;

DWORD_PTR dwFlags = D3DXSHADER_DEBUG;// | D3DXSHADER_FORCE_VS_SOFTWARE_NOOPT;// | D3DXSHADER_FORCE_PS_SOFTWARE_NOOPT;

if( FAILED( D3DXCreateEffectFromFile( m_device, m_absPath, NULL, NULL, dwFlags, NULL, &m_effect, &pBufferErrors ) ) )
DEBUG_INFO( DEBUG_FATAL, "Failed to create shader from file: %s\nReason: %s", name, pBufferErrors->GetBufferPointer() );
}

////////////////////////////////////////////////////////////////////////////////
// Shader::Destroy
//! DESCRIPTION
//!
//! \return void -
////////////////////////////////////////////////////////////////////////////////
void Shader::Destroy( )
{
if( m_effect )
{
m_effect->Release();
m_effect = NULL;
}
}

////////////////////////////////////////////////////////////////////////////////
// Shader::SetTechnique
//! Set the current rendering technique
//!
//! \param name -
//! \return void -
////////////////////////////////////////////////////////////////////////////////
bool Shader::SetTechnique( char * name )
{
if( !name )
return false;

if( !m_effect )
return false;

m_technique = m_effect->GetTechniqueByName( name );

if( FAILED( m_effect->ValidateTechnique( m_technique ) ) )
return false;

return SUCCEEDED( m_effect->SetTechnique( m_technique ) );
}

////////////////////////////////////////////////////////////////////////////////
// Shader::SendRenderData
//! Sends the default data to the shader
//!
//! \return void -
////////////////////////////////////////////////////////////////////////////////
void Shader::SendRenderData( D3DXMATRIX * world )
{
if( !m_application )
return;

if( !m_effect )
return;

Renderer * pDevice = m_application->GetRenderer( );

if( !pDevice )
return;

D3DXMATRIX* matWorld = world ? world : pDevice->GetWorldTransform( );
D3DXMATRIX* matView = pDevice->GetViewTransform( );
D3DXMATRIX* matProj = pDevice->GetProjectionTransform( );

D3DXMATRIX matWorldViewProj;
D3DXMatrixIdentity( &matWorldViewProj );

matWorldViewProj = (*matWorld * *matView * *matProj);

m_effect->SetMatrix( "gWorldMatrix", matWorld );
m_effect->SetMatrix( "gViewMatrix", matView );
m_effect->SetMatrix( "gProjMatrix", matProj );
m_effect->SetMatrix( "gWorldViewProj", &matWorldViewProj );
m_effect->SetVector( "gViewPos", &pDevice->GetViewVector( ) );
}

////////////////////////////////////////////////////////////////////////////////
// Shader::Begin
//! Begin the render
//!
//! \param * passes - pointer to variable to contain the number of passes
//! \return bool -
////////////////////////////////////////////////////////////////////////////////
bool Shader::Begin( unsigned int * passes )
{
unsigned int iPasses = 0;

if( !m_effect )
return false;

if( !m_technique )
return false;

if( FAILED( m_effect->Begin( &iPasses, NULL ) ) )
return false;

if( passes )
*passes = iPasses;

return true;
}

////////////////////////////////////////////////////////////////////////////////
// Shader::Pass
//! Start a render pass
//!
//! \param pass - the index of the pass number
//! \return bool -
////////////////////////////////////////////////////////////////////////////////
bool Shader::Pass( unsigned int pass )
{
if( !m_effect )
return false;

if( !m_technique )
return false;

return SUCCEEDED( m_effect->BeginPass( pass ) );
}

////////////////////////////////////////////////////////////////////////////////
// Shader::EndPass
//! End the shader pass
//!
//! \return bool -
////////////////////////////////////////////////////////////////////////////////
bool Shader::EndPass( )
{
if( !m_effect )
return false;

if( !m_technique )
return false;

return SUCCEEDED( m_effect->EndPass( ) );
}

////////////////////////////////////////////////////////////////////////////////
// Shader::End
//! End the render pass
//!
//! \return bool -
////////////////////////////////////////////////////////////////////////////////
bool Shader::End( )
{
if( !m_effect )
return false;

if( !m_technique )
return false;

return SUCCEEDED( m_effect->End( ) );
}


- Dump your transformation matrices from within PIX
- Grab a known good D3D sample program and dump the transformation matrices from that.

- Plug your transformation matrices into the the known good program (K).
1. Verify that K is rendering in PIX.
2. Verify that K now works in PIX, if it does, AT LEAST your VS is broken
3. If it doesn't, AT LEAST your transformation matrices generation is broken.

And the other way around...
- Plug the known good program's transformation matrices into your program
1. If your program now works, it means that AT LEAST your transformation matrices generation is incorrect
2. If your program is still broken, it means that AT LEAST your VS is incorrect

Or you could become really good at linear algebra (unlike me ) and be able to spot transform problems by looking at matrices

Other stuff..
- think it's mul(coord, mat) too... set some crazy translation (10000, 20000), debug the vertex in PIX, and make sure the matrix multiply makes sense
- verify that the matrix in PIX is the same as when you passed it in.
- set your world/view to identity and make sure your projection matrix is correct.

Thanks for all the replies everybody. It turns out that my shaders and matrices were fine all along, it was actually fog that was causing the the shader to render all the vertices the same colour as the fog. It looked like it wasn't drawing because I have fog set to the same colour as the Clear colour.

Any reason why fog would cause that?