TL;DR: skip the first 3 paragraphs.
Hi! I've been toying around with building a 2D engine from scratch using Direct3D 11 for about a week, but I feel I'm in pretty deep water now and I could use some help. I'm very new to D3D, and I never used D3D9 for example, so most of what I've been able to accomplish is from looking at RasterTek's tutorials, and reading the MSDN documentation. Since 2D has been the plan all the way, I've ignored (and completely forgotten about) the Z dimension, but I feel it could come in handy again now for the rendered order of sprites, as my current approach seems very convoluted and inefficient.
First off, I have a stack of Scene objects to control the game state, inspired by the ScreenManager sample from XNA a while back. The scenes are updated from the top down, and when a Scene is closed (such as a menu for example) it is popped off the stack so the focus drops down to the next Scene, which could be the actual game in this example. Pointers to these Scene objects are then sent to my render subsystem and stacked in reverse order, as these will need to be rendered from the bottom up. This part works, and the scenes are rendered in the correct order.
However, I would also like to be able to give different layers to the renderables in a Scene, such as setting text on top of sprites for menus. This is where it currently gets iffy. Currently, I have a std::map with the Scene pointer as its key, and then a std::set of Renderable objects as the value. This set is sorted by a functor that compares a Z index value set for each individual Renderable object. Lower indices are rendered below the higher ones. The good thing about this system is that it works; it does what it's supposed to. The bad thing is that it'll probably become incredibly slow once I get to a certain number of Renderables in the set, because of the logarithmic insert/removes. Since the set doesn't sort itself, it's very impractical to change a Renderable's Z value as well, because I then have to remove it and add it to the set again.
What struck me after quite a while of fighting with this problem, though, is... why am I reimplementing something that is already done natively by the graphics card in 3D applications? Instead of a set, why not just use a std::vector with the Renderables, set their Z position as well to represent their respective layering order, and let the graphics card handle the depth ordering of my vertices. So that's what I'm trying to do now, but I can't get it to work. At first I just tried to set the Z component of the position, but it just ignored that and rendered the Renderables in the order they were added to the vector instead. I hadn't set up a depth buffer, so I did that as well, but then I just get a blank screen. I'm too new with D3D to understand what's going on, and I'm not sure what information you guys might need to try and help me, so I'll just post what I think could be relevant.
The 2D orthographic view-projection matrix that's passed to my vertex shader:
XMMATRIX viewMatrix = XMMatrixIdentity();
XMMATRIX projectionMatrix = XMMatrixOrthographicOffCenterLH( 0.0f, resX, resY, 0.0f, minZ, maxZ );
XMMATRIX viewProjectionMatrix = XMMatrixMultiplyTranspose( viewMatrix, projectionMatrix );
XMStoreFloat4x4( &mViewProjectionMatrix, viewProjectionMatrix );
MinZ and MaxZ are currently set to -100.0f and 100.0f respectively, and Renderables are by default placed at Z 0.0f. ResX and ResY are the screen resolution, 1024x576 with the current defaults.
My viewport:
vp.Width = resX;
vp.Height = resY;
vp.MinDepth = 0.0f;
vp.MaxDepth = 1.0f;
vp.TopLeftX = 0.0f;
vp.TopLeftY = 0.0f;
My active rasterizer state:
rasterizerDescs[ 0 ].AntialiasedLineEnable = true;
rasterizerDescs[ 0 ].CullMode = D3D11_CULL_NONE;
rasterizerDescs[ 0 ].DepthBias = 0;
rasterizerDescs[ 0 ].DepthBiasClamp = 0.0f;
rasterizerDescs[ 0 ].DepthClipEnable = true;
rasterizerDescs[ 0 ].FillMode = D3D11_FILL_SOLID;
rasterizerDescs[ 0 ].FrontCounterClockwise = false;
rasterizerDescs[ 0 ].MultisampleEnable = true;
rasterizerDescs[ 0 ].ScissorEnable = false;
rasterizerDescs[ 0 ].SlopeScaledDepthBias = 0.0f;
My attempted, but non-working, depth stencil state/view:
D3D11_TEXTURE2D_DESC depthTextureDesc;
depthTextureDesc.BindFlags = D3D11_BIND_DEPTH_STENCIL;
depthTextureDesc.Height = backBufferDesc.Height;
depthTextureDesc.Width = backBufferDesc.Width;
depthTextureDesc.MipLevels = 1;
depthTextureDesc.ArraySize = 1;
depthTextureDesc.Format = DXGI_FORMAT_D32_FLOAT_S8X24_UINT;
depthTextureDesc.SampleDesc.Count = 1;
depthTextureDesc.SampleDesc.Quality = 0;
depthTextureDesc.Usage = D3D11_USAGE_DEFAULT;
depthTextureDesc.CPUAccessFlags = 0;
depthTextureDesc.MiscFlags = 0;
D3D11_DEPTH_STENCIL_DESC stencilDesc;
stencilDesc.BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
stencilDesc.BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_DECR;
stencilDesc.BackFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
stencilDesc.BackFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
stencilDesc.DepthEnable = true;
stencilDesc.DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ALL;
stencilDesc.DepthFunc = D3D11_COMPARISON_LESS;
stencilDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
stencilDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_INCR;
stencilDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
stencilDesc.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
stencilDesc.StencilEnable = true;
stencilDesc.StencilReadMask = 0xFF;
stencilDesc.StencilWriteMask = 0xFF;
D3D11_DEPTH_STENCIL_VIEW_DESC svd;
svd.Flags = 0;
svd.Format = DXGI_FORMAT_D32_FLOAT_S8X24_UINT;
svd.Texture2D.MipSlice = 0;
svd.ViewDimension = D3D11_DSV_DIMENSION_TEXTURE2D;
They're all created in the correct order by the device, and throw no errors. I've also set the depth stencil view together with the render target view in the OMSetRenderTargets function, and I do the ClearDepthStencilView together with the ClearRenderTargetView, but to no avail. Just a blank screen.
If you'd like to peek at more of the code, it's all on https://github.com/Wedvich/Untopia (since it's an ongoing project, there are some obsolete classes and code in there). The game loop is in the Game class, while the render calls are in the RenderSystem class.
So, does anyone have any ideas as to what I'm doing wrong? Why can't I get the depth to work properly for this?
Sorry for the rambling wall of text, but I'm new to this and I'm not sure how to formulate it into a concise question. Please ask if I need to clear up anything.
I've also attached a Graphics Debugger log (basically PIX for VS2012) with a frame from the failed depth test, where nothing renders. I'm not sure what to make out of it myself, but here it is: [attachment=13256:graphicslog.zip]
