• 12
• 10
• 10
• 13
• 10

# mouse click coords to screen coords

This topic is 4932 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

What I am trying to do is take the coordinates from a mouse click, from getCursorPos() and screenToClient(), and convert them to my 3d coordinates, so I can draw a ray into space and check its intersection with various objects. Can someone help with a general way to convert the pixel coordinates to on-screen 3d coordinates ? (the camera can move freely and might be pointing in an arbitrary direction.) Thanks, Nitzan

##### Share on other sites
What API are you using?

In DirectX there is a helper function D3DXVec3Unproject.

Call the function two times, with your origin MouseX,MouseY, 0.0f and 1.0f. This will give you two point to form a ray with for intersection tests.

I think for OpenGL there's something similar.

##### Share on other sites
When a 3D world coordinate is transformed in to a 2D projected coordinate is loses it's depth. Consequently, it is impossible to transform a 2D coordinate back in to where it was in the world before the transformation.

Generally, what you need to do is use your 2D coordinates to compute a pick ray is screen space. You then need to transform this screen space pick ray by the inverse of your view matrix to get your pick ray origin and direction in world coordinates. You would then use this ray in your intersection tests.

Such a function could look similar to the one below (this is not optomised):

D3DXVECTOR3 PickWorld( HWND Hwnd, int ScreenWidth, int ScreenHeight ){    D3DXVECTOR3					vPickRayDir;					// Pick ray direction	D3DXVECTOR3					vPickRayOrig;					// Pick ray origin	D3DXVECTOR3					v;								// Vector used in computation	D3DXMATRIX					matProj;						// Retrieved projection matrix	D3DXMATRIX					matView, m;						// Retrieved view and computation matrices	POINT						ptCursor;						// Cursor position	BOOL						bHit;							// Was a supplied mesh face hit	DWORD						dwFace;							// The hit face number	float						fBarry1, fBarry2, fDist;		// Barycentric coordinates in face and distance from ray origin			// Error check	if( m_Graphics->GetDeviceCOM() == NULL ){ return D3DXVECTOR3(0.0f,0.0f,0.0f); }	if( m_LevelMesh == NULL )				{ return D3DXVECTOR3(0.0f,0.0f,0.0f); }	// Get the projection matrix	m_Graphics->GetDeviceCOM()->GetTransform( D3DTS_PROJECTION, &matProj );	// Get the cursor position	GetCursorPos( &ptCursor );	// Change coordinates relative to the window which the level is rendered in	ScreenToClient( Hwnd, &ptCursor );	// Compute the pick ray in screen space	v.x =  ( ( ( 2.0f * ptCursor.x ) / ScreenWidth  ) - 1 ) / matProj._11;    v.y = -( ( ( 2.0f * ptCursor.y ) / ScreenHeight ) - 1 ) / matProj._22;    v.z =  1.0f;	// Get the inverse view matrix	m_Graphics->GetDeviceCOM()->GetTransform( D3DTS_VIEW, &matView );	D3DXMatrixInverse( &m, NULL, &matView );    // Transform the screen space pick ray in to 3D coordinates	vPickRayDir.x  = v.x*m._11 + v.y*m._21 + v.z*m._31;    vPickRayDir.y  = v.x*m._12 + v.y*m._22 + v.z*m._32;    vPickRayDir.z  = v.x*m._13 + v.y*m._23 + v.z*m._33;    vPickRayOrig.x = m._41;    vPickRayOrig.y = m._42;    vPickRayOrig.z = m._43;	// Collect the hit information	D3DXIntersect( m_LevelMesh->GetMesh(), &vPickRayOrig, &vPickRayDir, &bHit, &dwFace, &fBarry1, &fBarry2, &fDist, NULL, NULL );	// If a hit is acquired - store the details	if( bHit == TRUE )	{		// Normalise direction		D3DXVECTOR3 Output, Result;		D3DXVec3Normalize( &Output, &vPickRayDir );		// Work out position position		Result = vPickRayOrig +	(fDist*Output);			return Result;	}	else	{		return D3DXVECTOR3(0.0f,0.0f,0.0f);	}}

For a little clarification, in the code above m_LevelMesh->GetMesh() just returns a mesh of type ID3DXMesh*. This is an ordinary mesh that is centred around the origin and not transformed when rendered.

If you want to test this computed pick ray against objects placed arbirtarily in your world then you would need to transform the ray origin and direction by the inverse of the object's world position matrix before doing the intersection test with the object. This moves the ray in to the object's body space for testing.

[Edited by - __Daedalus__ on September 14, 2004 5:34:59 AM]

##### Share on other sites
Quote:
 Original post by EndurionWhat API are you using?

This is the DirectX forum [smile]

##### Share on other sites
Quote:
Original post by Coder
Quote:
 Original post by EndurionWhat API are you using?

This is the DirectX forum [smile]

Erm... yeah... (slaps myself with a spork)

##### Share on other sites
__Daedalus__,

the code you posted assumes the camera is always pointing in 0,0,1 direction right ?

How do I correctly convert the pixel coordinates if the camera is pointing in some random direction ?

Edit: I actually have another question. Since I am setting the persepctive angle to 45 degrees, why cant I simply figure out the ray by hand ? Shouldnt be too hard to figure out the correct angles at every point (if it's linear). Is it linear ? Is it a correct method to implement picking ?

According to my calculations, if the perspetive is 45 degrees horizontally, it should be 33.69 degrees vertically at 640x480. So at the outer edge, if I rotate the ray by 33.69 in the x axis and 45 degrees in the z axis won't it be pependicular to the screen ? Then I can find any point(s) in between.

Thanks,

Nitzan

[Edited by - nitzan on September 14, 2004 5:47:51 PM]

##### Share on other sites
Quote:
 Original post by nitzan__Daedalus__, the code you posted assumes the camera is always pointing in 0,0,1 direction right ? How do I correctly convert the pixel coordinates if the camera is pointing in some random direction ? Edit: I actually have another question. Since I am setting the persepctive angle to 45 degrees, why cant I simply figure out the ray by hand ? Shouldnt be too hard to figure out the correct angles at every point (if it's linear). Is it linear ? Is it a correct method to implement picking ? Thanks, Nitzan
The camera is pointing in any random direction. The camera's arbitrary translation, and rotation values are stored in the view matrix which you retrieve each time in the above function by calling Device->GetTransform( D3DTS_VIEW, &matView ).

Doing these calculations, the pick ray origin is the same position as the the camera (e.g. view matrix) in the world. And the ray direction is the screen space pick ray (a 3D vector built using your mouse coordinates and the projection matrix) transformed in relation to the (inverse) view matrix.

##### Share on other sites
So I implemented the code you posted Daedalus, up to D3DXIntersect. It seems to work fine but I dont understand something. vPickRayOrig always ends up being the camera location, meaning the ray seems to eminate from the center of the screen instead of where I actaully clicked. Also vPickRayDir is a 2 dimensional vector (its Z value is always zero) pointing from the center of the screen toward the point of click. So what I am guessing is that D3DXIntersect does some more work. Any idea how to get the exact point of click and the correct direction the pick ray is traveling at (I am using my own intersection routine with my own model format) ?

Once again:
vPickRayDir seems to be a 2d vector originating at the center of the screen and pointing toward the direction of the mouse click.
vPickRayOrig is always the camera location, meaning the center of the screen.

Thanks!

Nitzan

##### Share on other sites
that is right nitzan. the origin of the ray should be from the cameras location. I had the same problems recently with picking so i know what its like. i came up with a simple analogy to help understand it.

Look out of a window. the glass in the window is your screen, your eye is the camera and everything past the glass is you world. your finger acts as your mouse cursor.

now, choose something you want to pick. (preferable towards the left edge of the window to highlight my point better) say a tree/fencepost/car etc.

now, put your finger on the glass where you see the object you chose. now, your finger should be fully or partly obscuring the object. Now keep you finger there, and move your head to looking at your finger at a right angle to the glass. you will see that you finger (read cursor) is only about 50-100 pixels away from the screen edge.

but, the object you picked is far past the screens area to the left. (lets say, in screen coordinates (ie, relative to the glass/screen) -1000 pixels)

this is why we need to do all the ray stuff, because the object isnt really there, its just where we see it.

another analogy, is with the perspective. to mimic a lower perspective, move your eye further from the window, and vice versa with a higher perspective.

hope that helps you understand. but what is your aim for this? are you trying to select (pick) something in your world? if you are then i suggest using 'bounding volumes'. This way you do your 'checking for a hit' with the bounding volume, and not the mesh (as in daedalus's code above). This is more efficient, but once you have determined it 'hit' the bounding volume, you can then go onto checking which part of the mesh it hit. but i suspect you wont be needing to do this.