Jump to content
  • Advertisement
Sign in to follow this  
Husbj

DX11 RenderDoc (0.28) not properly capturing output

This topic is 864 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

First off I'm posting this here since the draw calls listed below are DX11. Feel free to move it if it rather belongs in the "APIs and Tools" section.

 

Long story short, I am trying to debug some post process operations (basically output texture = input texture + screen quad + pixel shader) using the latest public version of RenderDoc. I noticed that it would display a completely black (ie. zero-filled) render target output texture for each of my post processing steps, which didn't quite add up with what I was seeing in my actual program. As such I tried to isolate the issue and indeed it still occurs in a very simple case of a single downsample pass. The odd thing however is that in this simplified case I'm still getting the correct output in my program so somehow it appears that RenderDoc isn't getting the same results. The problem may still be with my code however as I am having some issues with more complex post process chains, and I'm just getting lucky with my simplified example.

I was therefore thinking that I would post the relevant draw calls I'm making and ask if there may be something I should add or move elsewhere.

My "normal" scene renders still display as intended with RenderDoc, however I tried to keep the state changes to a minimum for my post process effects so it feels like I could be missing something there. I cannot find anything that feels "missing" to myself though, hence my asking for another pair of eyes to have a look.

 

So, here is my ordered list of state changes and draw calls for the image resampling that seems to fail according to RenderDoc, yet works as intended within my application. It should also be noted that neither the debug layer, nor RenderDoc itself is reporting any actual errors, unbound shader resources, etc. - the render targets merely appear to have nothing at all drawn to them. Furthermore, the input texture also is displayed as all black by RenderDoc and seems the likeliest source of the problem; drawing a (0, 0, 0, 0) texture will naturally result in a (0, 0, 0, 0) render target:

RSSetViewports(); 					// Sets viewport to size of the output texture, which is half the size of input
// Unbind any lingering resources to ensure there'll be no input/output conflicts; this basically binds an array of null SRVs
VSSetShaderResources(null);
PSSetShaderResources(null);
CSSetUnorderedAccessViews(null);
// Actual frame start
OMSetRenderTargets(output);				// Binds the render target texture that has the viewport size
IASetPrimitiveTopology(TriangleStrip);	// Using a triangle strip so that the vertex shader can create a screen quad from 4 vertex id's
// Constant buffers would be bound here but the resample effect doesn't need any; it's just a pass-through single texture shader
// so in the current effect I'm not binding any.
IASetInputLayout(null);					// Null is set here since the vertex shader only reads the SV_VertexId system variable and no actual mesh data
VSSetShader(vsScreenQuad); 				// Simple screen quad building vertex shader. Passes SV_Position and Texcoord0 to the pixel shader stage.
PSSetShader(psResample); 				// Simple pass-through shader; uses texcoords from the VS to apply the texture bound to register t0.
RSSetState(null); 					// Uses the default rasterizer state. This is reported properly by RenderDoc.
OMSetBlendState(bs); 					// Sets a standard blend state
OMSetDepthStencilState(dss); 			        // Sets a standard depth-stencil state
PSSetSamplers(diffuseSampler); 			        // Sets a standard sampler state; UVW_CLAMP, MIN_MAG_LINEAR_MIP_POINT, no mip bias
PSSetShaderResources(input); 			        // Binds the input texture. RenderDoc displays this as all-zero ("black") from the get-go as well.
// Drawing
DrawInstanced(4, 1, 0, 0); 				// Draws 4 vertices; no vertex data is bound and the vertex shader builds the actual geometry based on SV_VertexId.
// Unbinding shader resources to avoid future input / output mismatches again. Superfluous since it is done at the beginning; only added in to be "100% sure".
VSSetShaderResources(null);
PSSetShaderResources(null);
OMSetRenderTargetsAndUnorderedAccessViews(nullptr);

// A normal scene draw then follows, ending in a Present() call, making RenderDoc consider it a capture-able frame, but that has no 
// direct relation to the issues with the non-drawing.
// It should be mentioned however that if the output (render target) from above is used to texture anything in the scene draw pass, 
// RenderDoc will NOT draw it and still considers the output texture to be all zeroes. It will however draw properly to my visible program and I can save 
// the render target image to disk in which case it also looks correctly downsampled. Perhaps the weirdest of all is that RenderDoc WILL display the texture 
// as properly applied in the thumbnail for the captured frame, however it does never feature in the frame itself.
// If another frame is captured later on in which the "output" texture is only bound as input, ie. the above resampling isn't carried out again in that frame, 
// it will display properly as a texture in RenderDoc.

Grateful for any ideas.

Edited by Husbjörn

Share this post


Link to post
Share on other sites
Advertisement

I don't see you're issuing a clear (which is a huge red flag unless you're doing it on purpose and know what you're doing).

Perhaps you need to enable RenderDoc's save initials setting.

 

RenderDoc also allows you to check the entire pipeline, see the outputs of the VS, and even debug the VS and PS shaders. Have you tried that?

There's also a pixel history log that will tell you why a pixel is of that colour (e.g. it was cleared, then set to red by pixel shader, then rejected a pixel shader due to depth buffer, etc)

Share this post


Link to post
Share on other sites

Likewise, reporting RenderDoc bugs will get the swiftest response if you report them directly to me on github :).

Do you have an example capture or runnable executable you can share to demonstrate the problem?

Share this post


Link to post
Share on other sites

I don't see you're issuing a clear (which is a huge red flag unless you're doing it on purpose and know what you're doing).

That is deliberately omitted; I'm rendering a full-sized quad and the input texture has no transparent pixels or other kind of colour blending so the previous RT contents are completely overwritten anyway.
I did reintroduce the clear for testing though; it makes no difference but it further confirms that the issue seems to be with the input texture being treated as blank rather than the output failing. A modification of the pixel shader to draw a constant colour regardless of the texture further attests to this.

 

RenderDoc also allows you to check the entire pipeline, see the outputs of the VS, and even debug the VS and PS shaders. Have you tried that?

Yes, the VS output is all correct. I'm not all too familiar with the capabilities of the pixel shader debugging functionality; from what I can tell it is sampling the input texture (it doesn't say outright what the sample results are though, although that should of course be the colour being written by the PS, ie. black/transparent according to RenderDoc).

 

There's also a pixel history log that will tell you why a pixel is of that colour (e.g. it was cleared, then set to red by pixel shader, then rejected a pixel shader due to depth buffer, etc)

Thank you! I was completely unaware of this, that's a handy feature.
That tells that the shader output is indeed float4(0, 0, 0, 0) for seemingly each and every pixel, so again this is reinforcing the idea that the input texture is considered blank for whatever reason.

 

 

Likewise, reporting RenderDoc bugs will get the swiftest response if you report them directly to me on github  :).

Yes, I thought of doing that, however I'm not quite sure it actually is a problem with RenderDoc itself rather than my own code.

I will try to shape something up to send you in the morning. Is there any advantage to sending an executable instead of a saved capture log? I should be able to do either.

Edited by Husbjörn

Share this post


Link to post
Share on other sites

Yes, I thought of doing that, however I'm not quite sure it actually is a problem with RenderDoc itself rather than my own code.

I will try to shape something up to send you in the morning. Is there any advantage to sending an executable instead of a saved capture log? I should be able to do either.

 

If there's reasonable doubt that it's a bug in RenderDoc I don't mind taking a look at it - oftentimes even when it ends up not being a RenderDoc bug I can point you in the right direction.

 

Sharing an executable is better than sending a capture because it gives me more ability to investigate - I might look at the capture, see the problem is 'oh, X is missing', but then need to go back to the executable to add logging or debug it to figure out where X got to during capture. Not everyone is comfortable sharing executables all the time though, so I help with whatever they can share.

Share this post


Link to post
Share on other sites

If there's reasonable doubt that it's a bug in RenderDoc I don't mind taking a look at it - oftentimes even when it ends up not being a RenderDoc bug I can point you in the right direction.

 

Sharing an executable is better than sending a capture because it gives me more ability to investigate - I might look at the capture, see the problem is 'oh, X is missing', but then need to go back to the executable to add logging or debug it to figure out where X got to during capture. Not everyone is comfortable sharing executables all the time though, so I help with whatever they can share.

Alright, thanks man, that's really kind of you.

I set about cleaning my D3D calls up a bit (can't show off how messy we usually are for public viewing now can we?  :wink: ) and now when recompiling the issues seem to have gone away... I tried to pinpoint what would have been the cause; mainly I was just removing double calls and moving some things around, however I haven't been able to find out what particular change solved it. The two main suspects were that I changed a depth-stencil state for one having depth testing disabled, since no depth-stencil buffer was bound for the resampling (it isn't needed for it after all) and that I was previously clearing my render targets before binding them to the output merger rather than after. When changed back to their prior states, none of these raised the issue again though. I will see if I have a backup with the problems intact such that I can pinpoint it.

 

Once I had simplified my main program enough (basically just drawing the resampled texture to a sprite) I also noticed however that that wouldn't get drawn by RenderDoc either.

The reason here turned out to be that RD doesn't seem to be aware of an InputLayout bound earlier than in the current frame. This wouldn't normally happen but when I was only drawing my 2D sprites that meant I only needed the one InputLayout and my engine is set up to only bind new ones if they're different from the one bound. Checking the "Save All Initials" box does not solve this, but maybe it isn't supposed to either? That's the only concrete thing I've found for now anyway; I will get back to you if I can find out the cause for the initial problems. The cause there seemed to be the input texture being considered blank, whereas the issue with the missing InputLayout had the input texture display as expected. Perhaps it can still be related such that the initial state of the texture wasn't stored in that case though? But I do recall it being bound on each frame since there were other things being drawn with other textures that would have to have it rebound so I'm not too sure.

 

In any case, you deserve commendations for not only providing this great tool for use, but also support free of charge. Cheers!

 

 

Edit: on a side note, what exactly is the meaning of the following and why is it considered a warning?

 

RENDERDOC:  [14:46:52]     dxgi_wrapped.cpp( 553) - Warning - Querying IDXGIObject for interface: GUID {7abb6563-02bc-47c4-8ef9-acc4795edbcf}

 

 

Sometimes I have to left-click the texture display in RenderDoc to make it update, have you tried that?

Yep, I'm aware of that, thanks for pointing it out though.

Edited by Husbjörn

Share this post


Link to post
Share on other sites

The reason here turned out to be that RD doesn't seem to be aware of an InputLayout bound earlier than in the current frame. This wouldn't normally happen but when I was only drawing my 2D sprites that meant I only needed the one InputLayout and my engine is set up to only bind new ones if they're different from the one bound. Checking the "Save All Initials" box does not solve this, but maybe it isn't supposed to either? That's the only concrete thing I've found for now anyway; I will get back to you if I can find out the cause for the initial problems. The cause there seemed to be the input texture being considered blank, whereas the issue with the missing InputLayout had the input texture display as expected. Perhaps it can still be related such that the initial state of the texture wasn't stored in that case though? But I do recall it being bound on each frame since there were other things being drawn with other textures that would have to have it rebound so I'm not too sure.


This case should be handled, although I imagine there can be edge cases that are buggy. RenderDoc keeps track of the current state at any point on the immediate context, and serialises that out at the start of the frame, when replaying it restores it again. Do you have an example that reproduces the missing tracking?

 

"Save All Initials" just overrides an optimisation heuristic that makes RenderDoc skip the initial frame contents of render targets that look like they're entirely overwritten in the frame.
 

Edit: on a side note, what exactly is the meaning of the following and why is it considered a warning?

RENDERDOC:  [14:46:52]     dxgi_wrapped.cpp( 553) - Warning - Querying IDXGIObject for interface: GUID {7abb6563-02bc-47c4-8ef9-acc4795edbcf}

 

 
This is mostly there so that I can catch any cases where objects have QueryInterface called for some GUID I'm not handling. This can lead to non-wrapped D3D handles leaking out that can lead to crashes or just bad behaviour. In this case it's some internal secret GUID that's queried inside D3D, so it's harmless.
 

 

Sometimes I have to left-click the texture display in RenderDoc to make it update, have you tried that?

Yep, I'm aware of that, thanks for pointing it out though.

 

 
I'm not aware!  :). Do you notice this happening consistently in some situations? or is it just every now and then something seems overly cached and needs a repaint.
 

In any case, you deserve commendations for not only providing this great tool for use, but also support free of charge. Cheers!


I'm glad it's useful, hopefully I can make it a little more bug-free :wink:.

Share this post


Link to post
Share on other sites

 

 

This case should be handled, although I imagine there can be edge cases that are buggy. RenderDoc keeps track of the current state at any point on the immediate context, and serialises that out at the start of the frame, when replaying it restores it again. Do you have an example that reproduces the missing tracking?

Hm, I see. Sure thing; here's a trimmed down example reproducing it: http://dx11.squarefantasy.net/InputLayoutExample.rar

(The resampling occurs in frame #2 in case you want to have a look at that as well; in this frame an InputLayout is set because it was set differently for the preceeding resampling procedure as well, so it should show the difference to any latter captured frame).

 

 

Edit: on a side note, what exactly is the meaning of the following and why is it considered a warning?

RENDERDOC:  [14:46:52]     dxgi_wrapped.cpp( 553) - Warning - Querying IDXGIObject for interface: GUID {7abb6563-02bc-47c4-8ef9-acc4795edbcf}

 
This is mostly there so that I can catch any cases where objects have QueryInterface called for some GUID I'm not handling. This can lead to non-wrapped D3D handles leaking out that can lead to crashes or just bad behaviour. In this case it's some internal secret GUID that's queried inside D3D, so it's harmless.

Ah, alright; I thought it was maybe complaining about me querying some interface (I know RenderDoc doesn't like me querying the debug interface while it's hooked) but couldn't find it. 

 

 

Sometimes I have to left-click the texture display in RenderDoc to make it update, have you tried that?

Yep, I'm aware of that, thanks for pointing it out though.

I'm not aware!  :). Do you notice this happening consistently in some situations? or is it just every now and then something seems overly cached and needs a repaint.

It happens every now and then but it isn't something I've thought much about such that it happens during conditions X and Y.

I would guess that it's an update event that doesn't trigger when changing settings through some particular UI component, or it could just be that a different open texture is considered active until it is manually clicked.

 

Speaking of this replaying functionality by the way, would there be any chance of providing support for changing states and / or buffer data from within RenderDoc to see how that would affect the follow-up graphics operations sometime in the future? I suspect this should be possible through injecting additional update calls / creating additional states and setting those instead. I think that would be a very nice tool to have available.

Share this post


Link to post
Share on other sites

Hm, I see. Sure thing; here's a trimmed down example reproducing it: http://dx11.squarefantasy.net/InputLayoutExample.rar
(The resampling occurs in frame #2 in case you want to have a look at that as well; in this frame an InputLayout is set because it was set differently for the preceeding resampling procedure as well, so it should show the difference to any latter captured frame).


Fixed and I've snuck that into the new v0.29 I was in the middle of preparing. Long story short, I recently moved InputLayouts so they were reference-tracked for inclusion into captures rather than just all being included. When I did that, I failed to mark that initial-state layout as referenced, so it got trimmed from the log.

This actually also exposed a bug where any executable with 'renderdoc' in the filename wouldn't capture due to an overly aggressive filter on which modules to hook. I'll fix that at some stage but it's unlikely to trip many people up.
 

Ah, alright; I thought it was maybe complaining about me querying some interface (I know RenderDoc doesn't like me querying the debug interface while it's hooked) but couldn't find it.


Which debug interface is that? Querying for ID3D11Debug should work just fine, is there another one?
 

It happens every now and then but it isn't something I've thought much about such that it happens during conditions X and Y.
I would guess that it's an update event that doesn't trigger when changing settings through some particular UI component, or it could just be that a different open texture is considered active until it is manually clicked.


Hmm, could be a few things I guess. If you do manage to spot a pattern file an issue on github and I'll look into it
 

Speaking of this replaying functionality by the way, would there be any chance of providing support for changing states and / or buffer data from within RenderDoc to see how that would affect the follow-up graphics operations sometime in the future? I suspect this should be possible through injecting additional update calls / creating additional states and setting those instead. I think that would be a very nice tool to have available.


Yes editing states and resource contents is a common request and one that I have on my roadmap. The implementation isn't too bad actually, but it's a question of how to display it in the UI - and with D3D11 when you edit a state does it just apply at that drawcall? or until the next time the state changes? or is it retroactive further back elsewhere in the frame, where the same state object is used. At different times you might want all three options, so it's a complex UX task to solve.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!