Help me understand how to hook DX application, to copy screen buffer

Started by
4 comments, last by Bacterius 11 years, 9 months ago
I'm developing application in C#, and i'm currently strugling in finding the general correct steps required to sucessfully hook DX function in order to send screen buffer to my main application for some processing.

Here is a list of what i think i should be doing so far:

1. Attach a custom C++ dll into target application, from my main application (written in C#)
2. DLL hooks some (Device Present and Reset) functions in d3d9.dll that is loaded in target application.
3. After even occurs, i need to send a callback event from the custom dll to my main application, containing the screen data

Can someone please correct me where i could have gone wrong and help me with point 3? I'm not expirerenced in dll callbacks, let alone not a C++ interop guru and im having a trouble understanding how i could possibly have a callback to my main application from a C++ dll loaded in another application.
Advertisement
Why don't you simply use detours.

Why don't you simply use detours.


Why would i use complete library only to install a hook to 2 functions? Its also not free for commercial and limited in express edition, lets not be ridicilous here.

Why would i use complete library only to install a hook to 2 functions? Its also not free for commercial and limited in express edition, lets not be ridicilous here.

Because its a lot simpler and safer then doing it manually. If your doing this for a commercial reason, I see no reason why you wouldn't benefit from buying it, and I am pretty sure the express version is all your going to need for what you specified. I also would question why your trying to do this? Typically this is the type of thing game hacks do, which I am willing to guess is exactly what your trying to do.

But a quick google search produced the following: http://spazzarama.co...ct3d-api-hooks/

I also would question why your trying to do this? Typically this is the type of thing game hacks do, which I am willing to guess is exactly what your trying to do.

But a quick google search produced the following: http://spazzarama.co...ct3d-api-hooks/


Copying screen buffer is not a hack, it does not change or automate any of the behaviour of the software, if doing that would be considered hack, then using things like fraps or any other screen recording software would be bannable offense.

I know about that spazzarma api hook, but its overly bloated with extra threads, and probably even changes running application by running CLR runtime host on it, and i dont want that. Also requires your assembly to be installed in GAC.
You can't callback from a DLL located in separate process, they belong in different memory spaces. Your callback will be located in your own process's memory space, whereas the DLL hooked into the game will be in the game's process's memory space. They can't communicate via native function pointers (callbacks).

You will need to use OS-provided functions to achieve inter-process communication at this point. An interesting solution is to use Windows messages to send "I have a frame ready" events (remember to use RegisterWindowMessage to get a system-wide message ID), and use a suitably sized shared memory section to dump the frame buffer in. Then in your main application, you monitor your message queue, and when you receive a message like that, you read the shared memory to get the screen data!

You will need to add synchronization however, or you might end up skipping frames. Obviously you can't send back messages to the game without hooking into its message loop, so instead you want to write some kind of header to shared memory, something along the lines of "ok I'm ready, dump the next frame" with the DLL hook waiting until it reads this. Or you might be happy with skipping frames (depending on what kind of post-processing you are doing, you may not be able to keep up with the game anyway).

Or just ditch messages altogether and use pipes+shared memory to communicate, etc... look up IPC (inter-process communication) and find something that can be done in the hook without too much hassle and overhead.

“If I understand the standard right it is legal and safe to do this but the resulting value could be anything.”

This topic is closed to new replies.

Advertisement