How I do it, is I color pick the scene. What I mean by that is that you render an off screen surface of only your buttons, each one a different solid color(randomly generated). Then on mouse down event, check what color the pixel is that your mouse is hovering over, and that is the button that is "clicked".
Edit: Granted, this method I propose is fairly lengthy, but it allows for picking ANY object in the scene, no matter what you are drawing. It is image based, so this method supports any number of objects on screen. In your case, if your buttons are always going to be in the same location, it would be easier to
just check the RECT coordinates of the buttons and see if your mouse is over or not.
Edit Again: Sorry! Just noticed you are writing in C#, so this code isn't applicable, but the concept is still the same =)
This is my implementation minus the actual drawing.
//Create offscreen surface to read surface data
IDirect3DSurface9* offScreenBuffer;
if(_d3dDevice->CreateOffscreenPlainSurface(_appWidth, _appHeight, D3DFMT_X8R8G8B8, D3DPOOL_SYSTEMMEM, &offScreenBuffer, 0) != D3D_OK)
ErrorMessenger::ReportMessage("Failed to create off screen surface!", __FILE__, __LINE__);
IDirect3DSurface9* source;
_d3dDevice->GetRenderTarget(0, &source);
if(_d3dDevice->GetRenderTargetData(source, offScreenBuffer) != D3D_OK)
ErrorMessenger::ReportMessage("Failed to GetRenderTargetData!", __FILE__, __LINE__);
//Lock surface data
D3DLOCKED_RECT surfaceData;
offScreenBuffer->LockRect(&surfaceData, 0, D3DLOCK_READONLY);
//Sample for pixel color to determine what object the mouse is colliding with
BYTE* bytePointer = (BYTE*)surfaceData.pBits;
//Get the index where the mouse is located
DWORD index = (mouseX * 4 + (mouseY * (surfaceData.Pitch)));
//Get the color
//X8R8G8B8 surface format
BYTE green = bytePointer[index];
BYTE blue = bytePointer[index + 1];
BYTE red = bytePointer[index + 2];
offScreenBuffer->UnlockRect();
//Find the object that belongs to the color
D3DXVECTOR4 searchColor = D3DXVECTOR4(((float)red / 255.0f), ((float)blue / 255.0f), ((float)green / 255.0f), 1.0f);
for(int i = 0; i < _worlds[_views[view].world].Size(); i++)
{
//Check in a range, color output from PS is not perfect
if(searchColor.x <= colors[i].x + 0.01f && searchColor.x >= colors[i].x - 0.01f &&
searchColor.y <= colors[i].y + 0.01f && searchColor.y >= colors[i].y - 0.01f &&
searchColor.z <= colors[i].z + 0.01f && searchColor.z >= colors[i].z - 0.01f)
{
return i;//Object found
}
}
//Mouse was not colliding with an object.
return NO_INDEX;