Problem with loading DLL / SetWindowsHookEx

Started by
14 comments, last by XTAL256 14 years ago
I am baffled as to why this doesn't work. I have created a DLL and am trying to load it using the code below.

/*** Called from my main executable "Window Detective.exe" ***/

HOOKPROC hookProc;
HINSTANCE hinstDLL;
HHOOK hhook;

hinstDLL = LoadLibrary(L"WD_Hook.dll");
hookProc = (HOOKPROC)GetProcAddress(hinstDLL, "hookProc");
hhook = SetWindowsHookEx(WH_GETMESSAGE, hookProc, hinstDLL,0);

// Send WM_NULL to all windows to force loading
// ...



/*** DLL code ***/

/// header file
#ifndef WD_HOOK_H
#define WD_HOOK_H

extern "C" {

#ifdef WD_HOOK_EXPORTS
  #define WD_HOOK_API __declspec(dllexport)
#else
  #define WD_HOOK_API __declspec(dllimport)
#endif


WD_HOOK_API LRESULT hookProc(int code, WPARAM wParam, LPARAM lParam);

}

#endif   // WD_HOOK_H

/// cpp file
#include "stdafx.h"
#include "Hook.h"

// Shared data
// Seen by both the instance of this DLL mapped into the remote
// process as well as the instance mapped into our exe
#pragma data_seg (".shared")
#pragma comment(linker,"/section:.shared,rws")
 HHOOK hook = NULL;
#pragma data_seg ()

BOOL APIENTRY DllMain(HMODULE module, DWORD reasonForCall, LPVOID reserved) {
    return TRUE;
}

// TODO: Catch messages here and do stuff with them
LRESULT hookProc(int code, WPARAM wParam, LPARAM lParam) {
    return CallNextHookEx(hook, code, wParam, lParam);
}


I have managed to get the DLL to load into my process (the one that calls this code), but it will not load into other processes. Now here is the thing i don't get. I use Smalltalk at work, so, as a test, i tried the above code in my Smalltalk IDE and it worked! It is essensially calling the same Windows functions, but for some reason it does not work when i do it in my project. I am using C++ with Qt (compiling with qmake and nmake) but i don't see how that could make a difference. My project is Unicode, whereas the Smalltalk VM is ASCII. But again, i don't think that would matter. I really can't give you any more information because i don't know what else i can try. Hopefully someone has an idea as to why this is not working...
[Window Detective] - Windows UI spy utility for programmers
Advertisement
What operating system are you using (both at home and at work)? What permissions does your program have?
If I put my 2 cents in and get a penny for my thoughts, where does my other penny go?
Actually, both cases were tested at work, in an XP virtual machine.
I also thought about permissions and i was going to try running as admin but i forgot. I just tried that now (at home) but it still didn't work. I'm not sure what other permissions the program could have, but i doubt they would be running with different permissions.
[Window Detective] - Windows UI spy utility for programmers
Hmmm ... It should work on XP, I was more concerned about the UAC in Vista/Win7 silently blocking your program from injecting.

I don't know if your program needs debug privileges to set a hook like that, but just in case here's a snippet (not mine, but I don't recall where I got it) I use before modifying other processes (it's required, at least, if you want to OpenProcess something with PROCESS_ALL_ACCESS in XP+):
void EnableDebugPriv() {    HANDLE hToken;    LUID luid;    TOKEN_PRIVILEGES tkp;    OpenProcessToken( GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &hToken );    LookupPrivilegeValue( NULL, SE_DEBUG_NAME, &luid );    tkp.PrivilegeCount = 1;    tkp.Privileges[0].Luid = luid;    tkp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;    AdjustTokenPrivileges( hToken, false, &tkp, sizeof( tkp ), NULL, NULL );    CloseHandle( hToken ); }

If I put my 2 cents in and get a penny for my thoughts, where does my other penny go?
Thanks for that code snippet Amrazek. I'm not sure where i should use it but it didn't work when i put it before the SetWindowsHook code or in the DLL itself.
I don't know what else to try.
[Window Detective] - Windows UI spy utility for programmers
How do you know your DLL isn't loading into the other process? Do you ever get your DLL_PROCESS_ATTACH handler hit (You can shove in an OutputDebugString and pick it up with DebugView).

Do you get anything interesting showing up in Process Monitor?
LRESULT hookProc(int code, WPARAM wParam, LPARAM lParam) {    return CallNextHookEx(hook, code, wParam, lParam);}


You forget the WINAPI/CALLBACK part on the declaration of hookProc, it won't cause the dll not to load but it wont endear it to the target process when it unbalances the stack. Unless you're compiling with stdcall as the default convention that is, if so, ignore me.

This, along with Steve's suggestion might help you figure out what is going wrong. Since you're injecting into every app, open notepad and target that for the least debug spew.
Thank you all for your help. I wouldn't know what to do without you [grin].

@Evil Steve: I am looking at the DLLs pane in Process Explorer to determine if it is loaded in other processes. When i tried in Smalltalk, i could see the DLL being injected into a windowed process as soon as it receives a window message (as the MSDN docs say).
I will also try the two things you suggested.

@adeyblue: Thanks for the link, I will have a look at it later. I will also use the WINAPI declaration.
[Window Detective] - Windows UI spy utility for programmers
I put some OutputDebugString's in DllMain and it did not show it being attached to any process other than my own. Here is a portion of the output if anyone is interested (my debug strings are the dll attach flags):
......'Window Detective.exe': Loaded 'C:\WINDOWS\system32\ieframe.dll''Window Detective.exe': Loaded 'C:\WINDOWS\system32\iertutil.dll''Window Detective.exe': Loaded 'C:\Documents and Settings\...\Window Detective\WD_Hook.dll', Symbols loaded.DLL_PROCESS_ATTACH'Window Detective.exe': Loaded 'C:\WINDOWS\system32\wdmaud.drv''Window Detective.exe': Loaded 'C:\WINDOWS\system32\wintrust.dll''Window Detective.exe': Loaded 'C:\WINDOWS\system32\crypt32.dll''Window Detective.exe': Loaded 'C:\WINDOWS\system32\msasn1.dll''Window Detective.exe': Loaded 'C:\WINDOWS\system32\imagehlp.dll''Window Detective.exe': Unloaded 'C:\WINDOWS\system32\wdmaud.drv''Window Detective.exe': Loaded 'C:\WINDOWS\system32\wdmaud.drv'DLL_THREAD_ATTACH'Window Detective.exe': Loaded 'C:\WINDOWS\system32\msacm32.drv''Window Detective.exe': Loaded 'C:\WINDOWS\system32\msacm32.dll''Window Detective.exe': Loaded 'C:\WINDOWS\system32\midimap.dll'DLL_THREAD_ATTACHDLL_THREAD_ATTACHDLL_THREAD_DETACHThe thread 'Win32 Thread' (0xd08) has exited with code 0 (0x0).DLL_THREAD_ATTACHDLL_THREAD_DETACHThe thread 'Win32 Thread' (0xb98) has exited with code 0 (0x0).......
[Window Detective] - Windows UI spy utility for programmers
Obviously the first thing i checked when it didn't work was the error code returned by GetLastError. And i'm sure it did not give my any errors when i checked.

But i just checked again now that i am at home and i get error 127 (ERROR_PROC_NOT_FOUND). I don't know why i didn't get this the first time, and actually, this is after i added CALLBACK to the function's declaration, so maybe that has something to do with it.

I just looked at the function in Dependency Walker and it is defined as "_HookProc@12". Why is that? I am using extern "C" when declaring it, so it is not a C++ function. And when i checked Dependency Walker last time (i think it was before i added CALLBACK), the function was just called "HookProc"
[Window Detective] - Windows UI spy utility for programmers

This topic is closed to new replies.

Advertisement