[D3D11] Overlay/Hooking Problems

Started by
4 comments, last by Necrolis 13 years, 1 month ago
Hey all. I'm currently working on a generic game 'overlay' (similar to the Steam or Xfire overlays, which I'm sure many of you are familiar with), and I'm having trouble testing my new D3D11 hooks. I'm currently hooking both device and swap chain creation, and all seems to be working as intended (indeed, rendering works if I inject my test overlay into one of the D3D11 tutorials), however I can't for the life of me figure out how to get the test triangle I'm trying to render to render on top of the game.

In D3D9 my test was easy, I simply hooked IDirect3DDevice9::EndScene, and used an ID3DXFont to draw some text on the screen. In D3D11 however (which I have ZERO experience with except for this overlay test) things are very different...

I'm currently trying to draw on-screen from my hook on IDXGISwapChain::Present, but I can never see anything on-screen... I assume there's something wrong with my rendering code, but I don't know enough about D3D to figure it out...

If anybody could look at my code and tell me what I'm doing wrong I'd greatly appreciate it. I don't actually need to know how to use D3D11, as I'm just going to use CEGUI once I can confirm everything is working as intended, however, I don't actually know enough to get my test working. :P

Relevant code (I know it's awful, but atm it's only a PoC):
http://code.google.c...11/Renderer.hpp
http://code.google.c...3D11/Hooker.hpp
http://code.google.c...3D11/Hooker.cpp
Note: In my IDXGISwapChain::Present hook I call "m_pKernel->OnFrame(*m_pRenderer)", this in turn calls Renderer::DrawText. Sorry if the cyclic calls are a little confusing. The entire source tree is available though if you want to poke around anywhere else and see exactly how everything works. Be forewarned though, the code is pretty awful in its current state.

Thanks all. If you need clarification on anything don't hesitate to ask.

P.S. If it matters, my 'real world' test case is World of Warcraft running in D3D11 mode. Also, sorry for what I can only assume is a relatively poorly written post, it's 5:10 AM here and I've been at it all night, so I'm tired as hell and about to go to bed. :)
Advertisement

Hey all. I'm currently working on a generic game 'overlay' (similar to the Steam or Xfire overlays, which I'm sure many of you are familiar with), and I'm having trouble testing my new D3D11 hooks. I'm currently hooking both device and swap chain creation, and all seems to be working as intended (indeed, rendering works if I inject my test overlay into one of the D3D11 tutorials), however I can't for the life of me figure out how to get the test triangle I'm trying to render to render on top of the game.

In D3D9 my test was easy, I simply hooked IDirect3DDevice9::EndScene, and used an ID3DXFont to draw some text on the screen. In D3D11 however (which I have ZERO experience with except for this overlay test) things are very different...

I'm currently trying to draw on-screen from my hook on IDXGISwapChain::Present, but I can never see anything on-screen... I assume there's something wrong with my rendering code, but I don't know enough about D3D to figure it out...

If anybody could look at my code and tell me what I'm doing wrong I'd greatly appreciate it. I don't actually need to know how to use D3D11, as I'm just going to use CEGUI once I can confirm everything is working as intended, however, I don't actually know enough to get my test working. :P

Relevant code (I know it's awful, but atm it's only a PoC):
http://code.google.c...11/Renderer.hpp
http://code.google.c...3D11/Hooker.hpp
http://code.google.c...3D11/Hooker.cpp
Note: In my IDXGISwapChain::Present hook I call "m_pKernel->OnFrame(*m_pRenderer)", this in turn calls Renderer::DrawText. Sorry if the cyclic calls are a little confusing. The entire source tree is available though if you want to poke around anywhere else and see exactly how everything works. Be forewarned though, the code is pretty awful in its current state.

Thanks all. If you need clarification on anything don't hesitate to ask.

P.S. If it matters, my 'real world' test case is World of Warcraft running in D3D11 mode. Also, sorry for what I can only assume is a relatively poorly written post, it's 5:10 AM here and I've been at it all night, so I'm tired as hell and about to go to bed. :)
Did you already verify that the hook was successful before trying to render? I mean simple things like changing your hook to print a message using OutputDebugStringA would be fine for testing if the hook was successful or not.

Also, you might be able to use PIX to verify whether your hook is doing something it's not supposed to or not.

[quote name='RaptorFactor' timestamp='1298139221' post='4776367']
Hey all. I'm currently working on a generic game 'overlay' (similar to the Steam or Xfire overlays, which I'm sure many of you are familiar with), and I'm having trouble testing my new D3D11 hooks. I'm currently hooking both device and swap chain creation, and all seems to be working as intended (indeed, rendering works if I inject my test overlay into one of the D3D11 tutorials), however I can't for the life of me figure out how to get the test triangle I'm trying to render to render on top of the game.

In D3D9 my test was easy, I simply hooked IDirect3DDevice9::EndScene, and used an ID3DXFont to draw some text on the screen. In D3D11 however (which I have ZERO experience with except for this overlay test) things are very different...

I'm currently trying to draw on-screen from my hook on IDXGISwapChain::Present, but I can never see anything on-screen... I assume there's something wrong with my rendering code, but I don't know enough about D3D to figure it out...

If anybody could look at my code and tell me what I'm doing wrong I'd greatly appreciate it. I don't actually need to know how to use D3D11, as I'm just going to use CEGUI once I can confirm everything is working as intended, however, I don't actually know enough to get my test working. :P

Relevant code (I know it's awful, but atm it's only a PoC):
http://code.google.c...11/Renderer.hpp
http://code.google.c...3D11/Hooker.hpp
http://code.google.c...3D11/Hooker.cpp
Note: In my IDXGISwapChain::Present hook I call "m_pKernel->OnFrame(*m_pRenderer)", this in turn calls Renderer::DrawText. Sorry if the cyclic calls are a little confusing. The entire source tree is available though if you want to poke around anywhere else and see exactly how everything works. Be forewarned though, the code is pretty awful in its current state.

Thanks all. If you need clarification on anything don't hesitate to ask.

P.S. If it matters, my 'real world' test case is World of Warcraft running in D3D11 mode. Also, sorry for what I can only assume is a relatively poorly written post, it's 5:10 AM here and I've been at it all night, so I'm tired as hell and about to go to bed. :)
Did you already verify that the hook was successful before trying to render? I mean simple things like changing your hook to print a message using OutputDebugStringA would be fine for testing if the hook was successful or not.

Also, you might be able to use PIX to verify whether your hook is doing something it's not supposed to or not.
[/quote]

I am 100% certain the hook is being called. Confirmed with a debugger and debut output.

The problem is not that the hook is not being called, the problem is that my D3D11 knowledge is effectively non-existent and I don't know enough to write the code to draw something on screen just to check that things are being called in the correct sequence.
[font="Monaco,"]if I understand the structure of your hook correctly ([s]its a little hard without knowing what Kernel::OnFrame does, though I'd assume thats where your doing is done, else your just missing drawing :P[/s] ok, found its implementation, nvm):[/font]
[color="#000000"] m_pDeviceContext[color="#666600"]->[color="#660066"]IASetVertexBuffers[color="#666600"]( [color="#006666"]0[color="#666600"], [color="#006666"]1[color="#666600"], [color="#666600"]&[color="#000000"]m_pVertexBuffer[color="#666600"], [color="#666600"]&[color="#000000"]stride[color="#666600"], [color="#666600"]&[color="#000000"]offset [color="#666600"]);[/quote]that line should be in your render call, along with the topology setting and render state setting, not in your init, because remember, the engine its self is doing work as well, so it'll overwrite what you set, unless you create your own deferred context to write stuff to (which might not be a bad idea).

[font="Monaco,"]if I understand the structure of your hook correctly ([s]its a little hard without knowing what Kernel::OnFrame does, though I'd assume thats where your doing is done, else your just missing drawing :P[/s] ok, found its implementation, nvm):[/font]
[color="#000000"] m_pDeviceContext[color="#666600"]->[color="#660066"]IASetVertexBuffers[color="#666600"]( [color="#006666"]0[color="#666600"], [color="#006666"]1[color="#666600"], [color="#666600"]&[color="#000000"]m_pVertexBuffer[color="#666600"], [color="#666600"]&[color="#000000"]stride[color="#666600"], [color="#666600"]&[color="#000000"]offset [color="#666600"]);
that line should be in your render call, along with the topology setting and render state setting, not in your init, because remember, the engine its self is doing work as well, so it'll overwrite what you set, unless you create your own deferred context to write stuff to (which might not be a bad idea).
[/quote]

Hey, thanks for the help, that kind-of worked. The triangle rendered on-screen whilst the game was still loading its resources etc and there was just a black screen for a few seconds, during that time my triangle was displayed, after that however, once the login screen appeared I could no longer see my triangle.

I assume the game is doing something with the render states or whatever that's causing my triangle not to be drawn on top, however I can't figure out what to do to override that. Any tips?

EDIT:


Weird, it only seemed to do it that one time, now I can't get it to render again whilst it's loading, just a black screen again. I assume I just got 'lucky' that one time and a certain 'window' during the load process was longer than normal and allowed me to get a glimpse of my triangle or something...

Anyway, still looking for a fix/workaround. Help would be hugely appreciated caus I have nfi what I'm doing when it comes to D3D. Lol. Thanks in advance,
might be that your z-depth is putting it behind every thing, try rendering it in 2d mode (aka orthogonally), see if it shows up
EDIT: just thought, you can also try disabling depth testing as well

This topic is closed to new replies.

Advertisement