I am currently working on a small deferred rendering engine for this semester's coursework assignment. The code compiles fine and runs in Visual Studio without any problems except no output to the screen and that's where the problems start.

The deferred rendering system is nothing special it is basically the first draft of the algorithms presented in "Practical Rendering with DirectX 11" [2011 by Jason Zink, Matt Pettineo, Jack Hoxley]. No optimization like Attribute packing etc. are used. Only a simple Lighting system is used (no shadow).

The problems seems to be the fullscreen-quad(simply inefficient fullscreen-quads used for lighting passes) or a general problem. Since I get no errors in the debug log I tried to use PIX and PerfStudio to get some more information on the gpu side. Unfortunately PIX and PerfStudio are before the first frame with this error:

Invalid allocation size: 4294967295 bytes

So for whatever reason it seems to allocate some space with -1 bytes. Awkwardly everything is fine in VisualStudio debugging... and if I attach a debugger to the PIX process and break when the error happens I land in a debuggers header file.

I just started using DirectX with prior OpenGL experience, so I hope I did not something generally wrong. I used the executable that was output by the compiler in debug mode.

To avoid general logic mistakes, here is roughly what I currently do:

1. SetDepthStencilState (with DepthTest enabled)
2. clear all RenderTargetViews (was unsure about gBuffer but gBuffer is being cleared as well currently) and DepthStencilBuffer
3. bind gBuffer and DepthStencilBuffer
4. render Geometry
5. disable DepthTest
6. bind Backbuffer
7. render all lights with the associated shader (since I am using the effects framework I set the BlendState in the shader)
8. render CEGUI (works fine even if rest doesn't output anything)
9. present()

The lights are as already mentioned fullscreen quads. The lighting technique is simply passing the position through, so the quads vertices are in the range [-1, 1].

If you need any additional informations let me know.

Thanks, Nik

P.S. sorry for the bad english...


EDIT:

For further informations: The vertices and indices of the fullscreen quad

glm::vec3 vertices[] =    {		glm::vec3(-1.0f, -1.0f,  1.0f),		glm::vec3(-1.0f,  1.0f,  1.0f),		glm::vec3( 1.0f, -1.0f,  1.0f),		glm::vec3( 1.0f,  1.0f,  1.0f),    };	    UINT indices[] = { 0, 3, 2, 2, 0, 1 };

And a rough walkthough the code:

// before geometry pass
        m_d3dImmediateContext->RSSetState(m_RasterState);
	m_d3dImmediateContext->OMSetDepthStencilState(m_GeometryDepthStencilState, 1);
	m_d3dImmediateContext->ClearRenderTargetView(m_RenderTargetView, reinterpret_cast<const float*>(&clearColor));
	m_d3dImmediateContext->ClearRenderTargetView(m_gBuffer[0], reinterpret_cast<const float*>(&clearColor));
	m_d3dImmediateContext->ClearRenderTargetView(m_gBuffer[1], reinterpret_cast<const float*>(&clearColor));
	m_d3dImmediateContext->ClearRenderTargetView(m_gBuffer[2], reinterpret_cast<const float*>(&clearColor));
	m_d3dImmediateContext->ClearRenderTargetView(m_gBuffer[3], reinterpret_cast<const float*>(&clearColor));
	m_d3dImmediateContext->ClearDepthStencilView(m_DepthStencilView, D3D11_CLEAR_DEPTH|D3D11_CLEAR_STENCIL, 1.0f, 0);
	m_d3dImmediateContext->OMSetRenderTargets(4, m_gBuffer, m_DepthStencilView);

// before lighting pass
        m_d3dImmediateContext->OMSetDepthStencilState(m_LightingDepthStencilState, 1);
	m_d3dImmediateContext->OMSetRenderTargets(1, &m_RenderTargetView, m_DepthStencilView);
	DXLightingShader->enable();

// DXLightingShader::enable (the static cast is necessary because the engine supports opengl and directx this is my dirty way
        static_cast<SDXRenderInfo*>(g_RenderInfo)->context->IASetInputLayout(m_InputLayout);
	static_cast<SDXRenderInfo*>(g_RenderInfo)->context->IASetPrimitiveTopology(D3D11_PRIMITIVE_TOPOLOGY_TRIANGLELIST);
	m_fxNormalMap->SetResource(m_NormalView);
	m_fxDiffuseMap->SetResource(m_DiffuseView);
	m_fxSpecularMap->SetResource(m_SpecularView);
	m_fxPositionMap->SetResource(m_PositionView);
	m_fxCameraPos->SetFloatVector(Camera->getPosition());

// depending on light-type this is how it is drawn
        for(UINT p = 0; p < m_DirectionalLightDesc.Passes; ++p)
	{
		m_DirectionalLight->GetPassByIndex(p)->Apply(0, static_cast<SDXRenderInfo*>(g_RenderInfo)->context);
		static_cast<SDXRenderInfo*>(g_RenderInfo)->context->DrawIndexed(6, 0, 0);
	}

// present function called after light passes
        DXLightingShader->disable();
	CEGUI::System::getSingleton().renderGUI();
	HR(m_SwapChain->Present(0, 0));

// DXLightingShader::disable
        m_fxNormalMap->SetResource(NULL);
	m_fxDiffuseMap->SetResource(NULL);
	m_fxSpecularMap->SetResource(NULL);
	m_fxPositionMap->SetResource(NULL);
	m_DirectionalLight->GetPassByIndex(0)->Apply(0, static_cast<SDXRenderInfo*>(g_RenderInfo)->context);

If you need more information or some details of the shader implementation let me know

Invalid allocation size: 4294967295 bytes

What makes you think this has anything to do with your deferred rendering setup? It could simply be an uninitialized variable somewhere else in your code. What header file do you break into?

Thanks for the tip. I am currently in chrismas stress but I am trying to debug the application again now.

So I figured out what the problem was, it was quite simple actually... Visual Studio has a different runtime environment, so pix wasn't able to find some files...

I am investigating why it is not rendering now, I'll hopefully come back with more informations later...

So I started debugging a frame with pix:

The 4 gBuffer textures are successfully being rendered.

The problem seems to be the fullscreen quad:

Since in the viewport output there is only a whiteline it seems to get discarded?

With z < 0 you're out of clipspace already. And a w of 0 will even send them to infinity. Can you show us that vertex shader code (and the transformation matrix values involved, if available).

Sure, sorry for delayed reply chrismas time...

for what ever reason the z value is always -1.0 also the preVS value...

but this is the vertex data in the associated buffer:

As already stated out previously I am new to directx, so is there anything that can influence how vertex data is interpreted since the preVS value is also -1?

Ok just noticed I was setting last value to 0.0 and not 1.0...
The problem now is I changed the z value of the input vertices to 0.0f no change, I changed the shader to set the z value to 0.0f I get some output, still not the right shaded cube but it seems to be a problem with my lighting code...

But this would be kind of a dirty fix, so why is the Z value being set to -1...?

Ah, already further, so yeah, your new vertex shader is probably fine. You can even do

float4 VSMain(in float4 Position : POSITION) : SV_Position
{
    return Position;
}

since w = 1 will happen automatically. This is the rare occasion where signature and layout may differ.

But your problem is elsewhere. PreVS means either your buffer init/update or your input layout is off - or you've bound the wrong buffer(s).

Thanks alot indeed I forgot to set the VertexBuffer and since the vertices of my cube are quite similar, I didn't notice that. Can't believe I haven't noticed that...

The only thing left now is a bug in my lighting code, some surfaces stay black... The scene currently uses a single directional light for testing!

float3 CalcLighting(in float3 normal, in float3 position, in float3 diffuseAlbedo, in float3 specularAlbedo, in float specularPower, uniform int gLightingMode) 
{  
float3 L = 0; 
float attenuation = 1.0f; 
if (gLightingMode == POINTLIGHT || gLightingMode == SPOTLIGHT)
{
L = LightPos - position; 
float dist = length(L); 
attenuation = max(0, 1.0f - (dist / LightRange.x)); 
L /= dist;
}
else if (gLightingMode == DIRECTIONALLIGHT)
{
L = -LightDirection; }if (gLightingMode == SPOTLIGHT)
{
float3 L2 = LightDirection; 
float rho = dot(-L, L2); 
attenuation *= saturate((rho - SpotlightAngles.y) / (SpotlightAngles.x - SpotlightAngles.y)); 
}
float nDotL = saturate(dot(normal, L)); 
float3 diffuse = nDotL * LightColor * diffuseAlbedo;
float3 V = CameraPos - position; 
float3 H = normalize(L + V);
float3 specular = pow(saturate(dot(normal, H)), specularPower) * LightColor * specularAlbedo.xyz * nDotL; 
return (diffuse + specular) * attenuation;
}

The basic algorithm is basically out of a book so I assumed there would be nothing wrong the only thing I changed is using if and a uniform to embed it in an effects file.

The Buffers are filled with informations and there is nothing missing, these are values of a black pixel that is supposed to have some color:


EDIT:

Lighting Shader Code on pastebin http://pastebin.com/8gbYBLCX, because the code tags seem to be buggy currently either escaping html as well or completly breaking formating

What color do you expect the surface to be? The vector L / -LightDirection (unnormalized) hits the surface orthogonal to its normal, so nDotL equals zero. Since both the diffuse and the specular coefficient are multiplied by nDotL the surface reflects no light. If you don't want to fade specular hi-lights with the light's incidence, you should remove the last '* nDotL'.