Mouse picking slightly inaccurate

Started by
4 comments, last by Duncanf 18 years, 5 months ago
Ello all, I've got some problems with picking, I've got it working but all my picks are slightly off target. The picking seems to work perfectly from the window coordinates 0,0 but as soon as I try anywhere else the actual location of the pointer and the location of the ray hit are increasingly wrong - worst in the bottom right of the window. Here's the code I'm using:

//SetCapture called in message handler
if( GetCapture() )
{
POINT ptCursor;
GetCursorPos( &ptCursor );
ScreenToClient( windowHandle, &ptCursor );

D3DXVECTOR3 v;
v.x =  ( ( ( 2.0f * ptCursor.x ) / winWidth  ) - 1 ) / projection._11;
v.y = -( ( ( 2.0f * ptCursor.y ) / winHeight ) - 1 ) / projection._22;
v.z =  1.0f;

D3DXMATRIX m;
D3DXMatrixInverse( &m, NULL, &view );

rayFromMouse.x  = v.x*m._11 + v.y*m._21 + v.z*m._31;
rayFromMouse.y  = v.x*m._12 + v.y*m._22 + v.z*m._32;
rayFromMouse.z  = v.x*m._13 + v.y*m._23 + v.z*m._33;
D3DXVec3Normalize(&rayFromMouse,&rayFromMouse);
rayFromMousePos = camera->getPosition();

SetCapture(0);
}

Does anyone know what I'm doing wrong? Cheers for any help, Duncan
Advertisement
This happend to me when my client space is smaller than I thought.

Repro of problem:

1. Window was 640x480.
2. Client area of window is slightly smaller than 640x480 due to title bar, etc.
3. Created D3D surface using the default values for the window. (which uses the client area size).
4. My mouse positioning calculations used 640x480 ans the size of the surface and not the actual client size.

Solution:
In step 3 I created the D3D surface to be of the proper size, 640x480, and everything lined up again.
Are you running in fullscreen or windowed mode? If your running in windowed mode, are you sure the actual drawing area is what you think it is? I mean the window frame and titlebar etc takes away some of the actual drawing space. That might cause a small offset.

This was my problem, and took about a week to finally figure out. doh.
Cheers for the replies, I've tried getting the exact (windowed) viewport size by getting the backbuffer and using the width and height from there - but it's still inaccurate. How are you getting your values of width and height?
To make sure the client area is the same size as the back buffer, use AdjustWindowRect to calculate the window rectangle that you pass to CreateWindow.

Thanks, that works perfectly. For anyone who searches the forum and needs a code sample:

RECT rect = { 0, 0, eng->winWidth, eng->winHeight };if (eng->windowed) AdjustWindowRectEx(&rect, WS_CAPTION | WS_SYSMENU, false, WS_EX_APPWINDOW);	//try to create a window using classeng->windowHandle = ::CreateWindow("DuncEngine", title, // class name, title				WS_EX_TOPMOST,	//style				0, 0, rect.right-rect.left, rect.bottom-rect.top,	//x,y,w,h				0,				//no parent				0,				//no menu 				instance,			//instance given				0);				//no extra mem 

This topic is closed to new replies.

Advertisement