Jump to content
  • Advertisement
Sign in to follow this  
  • entries
    557
  • comments
    1237
  • views
    423365

Untitled

Sign in to follow this  
Evil Steve

78 views

Yay, I got to play with virtual function tables today at lunch. Relevant thread.

What I found (VC8 specific probably, although it may work on other compilers):

  • The vtable pointer is held in the first 4 bytes of a class with virtual functions. That means you can cast the class to a size_t pointer, and dereference it to get the address of the vtable as a size_t.
    The vtable is an array of addresses, one entry per virtual function and is ordered in by the order the functions appear in the class. I'm not sure where a virtual destructor comes in, I didn't bother checking - but I expect it's the same as a normal func (Position defined by the location in the class header).

  • The vtable is write protected, so you need to use VirtualProtectEx to enable write access on it.

  • You need to use a naked stub function to call a "normal" C function to get around the calling convention crap.


Overall, nothing particularly mind blowing. Anyway, here's the code I ended up with (To hook the SetRenderState() call):

HRESULT MySetRenderState(D3DRENDERSTATETYPE State, DWORD Value)
{
// Do whatever you want here
}

__declspec(naked) HRESULT stub(D3DRENDERSTATETYPE State, DWORD Value)
{
__asm
{
push ebp;
mov ebp, esp;
sub esp, 0Ch;
}

MySetRenderState(State, Value);

__asm
{
mov esp, ebp;
pop ebp;
ret 0Ch;
}
}

// Grab the vtable and remove write protection on it
size_t nEntry = 57;
size_t* pVTable = (size_t*)*(size_t*)m_pDevice;
DWORD dwOldProt;
VirtualProtectEx(GetCurrentProcess(), &pVTable[nEntry], sizeof(size_t), PAGE_EXECUTE_READWRITE, &dwOldProt);

// Thunk the vtable so SetRenderState() calls "stub" instead
pVTable[nEntry] = (size_t)&stub;

// Test the new call
m_pDevice->SetRenderState(D3DRS_ZENABLE, 1);




Joy joy joy. Anyway, back to work for me...
Sign in to follow this  


0 Comments


Recommended Comments

There are no comments to display.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!