What I found (VC8 specific probably, although it may work on other compilers):
- The vtable pointer is held in the first 4 bytes of a class with virtual functions. That means you can cast the class to a size_t pointer, and dereference it to get the address of the vtable as a size_t.
The vtable is an array of addresses, one entry per virtual function and is ordered in by the order the functions appear in the class. I'm not sure where a virtual destructor comes in, I didn't bother checking - but I expect it's the same as a normal func (Position defined by the location in the class header).
- The vtable is write protected, so you need to use VirtualProtectEx to enable write access on it.
- You need to use a naked stub function to call a "normal" C function to get around the calling convention crap.
Overall, nothing particularly mind blowing. Anyway, here's the code I ended up with (To hook the SetRenderState() call):
HRESULT MySetRenderState(D3DRENDERSTATETYPE State, DWORD Value)
// Do whatever you want here
__declspec(naked) HRESULT stub(D3DRENDERSTATETYPE State, DWORD Value)
mov ebp, esp;
sub esp, 0Ch;
mov esp, ebp;
// Grab the vtable and remove write protection on it
size_t nEntry = 57;
size_t* pVTable = (size_t*)*(size_t*)m_pDevice;
VirtualProtectEx(GetCurrentProcess(), &pVTable[nEntry], sizeof(size_t), PAGE_EXECUTE_READWRITE, &dwOldProt);
// Thunk the vtable so SetRenderState() calls "stub" instead
pVTable[nEntry] = (size_t)&stub;
// Test the new call
Joy joy joy. Anyway, back to work for me...