How is it possible for the output register in a pixel shader to differ from PS output?

Started by
15 comments, last by relaxok 12 years, 1 month ago
Basically, the topic title is the question... any ideas?

here's the proof:

pixelshader_wtf.png
Advertisement
First thought - is any kind of blend mode enabled?

Also, what kind of render-target is that?

First thought - is any kind of blend mode enabled?

Also, what kind of render-target is that?


I have a blend state with alpha enabled most of the time, but switch BlendEnable to False for this terrain drawing - and it's just a basic render target setup like in most boilerplate, do you mean some specific parameter it was created with?

The weird thing is it's not all pixels - some pixel shader output is absolutely correct (and the debugger register will match what it says the pixel shader output is in PIX), but then the pixel right next to it will be almost black despite having a similar color in the output register - like the one i pasted.

This is a video where you can see the wrong pixels as a sort of artifact:

http://bh.polpo.org/texture_wtf.wmv
Can you post the HLSL?

Can you post the HLSL?


Here is a simplified version with a bunch of unused and unrelated stuff removed - i used this version with the same problematic result (random black pixels).

http://codepad.org/WAAklQ2G

I will soon be turning the ugly tons-o-texture-registers thing into another Texture2DArray but for now need those helper functions to return the correct one.

My confusion/issue still arises from the fact that I was under the impression the output register o0 is exactly the same as the 'pixel shader output' in PIX. It not being the same kind of breaks my brain/world. So there may very well be an issue with what I'm doing in the shader, but the output register looks correct.
You are comparing floating points without any epsilon, you might wanna reconsider that
Can you show in PIX what the OutputMerger state is? Perhaps there is something happening between your pixel shader and the render target writing that you haven't noticed before...

Can you show in PIX what the OutputMerger state is? Perhaps there is something happening between your pixel shader and the render target writing that you haven't noticed before...


Okay, however isn't the Pixel Shader Output unaffected by the OutputMerger, and post OutputMerger color is shown by the 'Final framebuffer color'?

I'm now drawing JUST the base opaque texture sampled, so no alpha blending at all.. and still getting the following behavior:

pixelshader_wtf4.png


Here are the OutputMerger infos:

pixelshader_wtf4b.png

pixelshader_wtf4c.png
A little update on this. I have no idea whats up with the pixel shader output not matching the output register, probably just a PIX bug in the debugger values.

However, I have narrowed down the problem to how the StructuredBuffer is being acessed with the meshNum from the input structure as an index.

I have done a test where I made sure the meshNum was always the same number when writing the vertices to the VB -- 251 in this case, since it was one of the messed up ones e.g. in that video I posted.

However.., amazingly accessing that StructuredBuffer TexIndices, with TexIndices[input.meshNum] returns different things depending on the pixel?? You can see it's almost random corruption in the video I linked above. If i FORCE it to index with the value 251 in the shader, as in specifically calling TexIndices[251] it will always be correct. So somehow the input.meshNum gets altered in the registers in some random fashion. It is NOT the StructuredBuffer itself getting corrupted because like I said it's fine if i index into it with a constant. Even on the pixels it's messed up on, the debugger shows 251 for the meshNum it's going to index with, as does the vertex shader which just copies that value, so it makes no sense.

I really think this must be some sort of dx11 bug or else some very weird behavior from this combination of techniques. I have wrestled with it all week and this is probably the closest I can get to determning the problem without someone way smarter than me suggesting things to try.
One other tidbit of information.. i get rid of 80% of the problem if i force output.meshNum to 251 in the vertex shader! So it's getting corrupted in the vertex shader as well. If I debug any of the vertices in a PIX frame capture, the meshNum is 251 in all of them even without doing that, but it's like PIX doesn't know what's up.

The interesting thing is is that with 80% of the problem removed, the corruption is much more infrequent - in fact, when i tried to fraps it and the framerate went down to 60 fps, I couldn't get it to happen - but it happens every 0.25 second or so at 1000 fps.

Granted, this wasn't really 'getting rid' of the problem, because in the end I can't force a value, it needs to be the one it gets from the vertex format.

I wish I could take more video - but anyway the video I posted is pretty similar to what is happening.

This topic is closed to new replies.

Advertisement