# mouse click coords to screen coords

This topic is 4837 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

What I am trying to do is take the coordinates from a mouse click, from getCursorPos() and screenToClient(), and convert them to my 3d coordinates, so I can draw a ray into space and check its intersection with various objects. Can someone help with a general way to convert the pixel coordinates to on-screen 3d coordinates ? (the camera can move freely and might be pointing in an arbitrary direction.) Thanks, Nitzan

##### Share on other sites
What API are you using?

In DirectX there is a helper function D3DXVec3Unproject.

Call the function two times, with your origin MouseX,MouseY, 0.0f and 1.0f. This will give you two point to form a ray with for intersection tests.

I think for OpenGL there's something similar.

##### Share on other sites
When a 3D world coordinate is transformed in to a 2D projected coordinate is loses it's depth. Consequently, it is impossible to transform a 2D coordinate back in to where it was in the world before the transformation.

Generally, what you need to do is use your 2D coordinates to compute a pick ray is screen space. You then need to transform this screen space pick ray by the inverse of your view matrix to get your pick ray origin and direction in world coordinates. You would then use this ray in your intersection tests.

Such a function could look similar to the one below (this is not optomised):

D3DXVECTOR3 PickWorld( HWND Hwnd, int ScreenWidth, int ScreenHeight ){    D3DXVECTOR3					vPickRayDir;					// Pick ray direction	D3DXVECTOR3					vPickRayOrig;					// Pick ray origin	D3DXVECTOR3					v;								// Vector used in computation	D3DXMATRIX					matProj;						// Retrieved projection matrix	D3DXMATRIX					matView, m;						// Retrieved view and computation matrices	POINT						ptCursor;						// Cursor position	BOOL						bHit;							// Was a supplied mesh face hit	DWORD						dwFace;							// The hit face number	float						fBarry1, fBarry2, fDist;		// Barycentric coordinates in face and distance from ray origin			// Error check	if( m_Graphics->GetDeviceCOM() == NULL ){ return D3DXVECTOR3(0.0f,0.0f,0.0f); }	if( m_LevelMesh == NULL )				{ return D3DXVECTOR3(0.0f,0.0f,0.0f); }	// Get the projection matrix	m_Graphics->GetDeviceCOM()->GetTransform( D3DTS_PROJECTION, &matProj );	// Get the cursor position	GetCursorPos( &ptCursor );	// Change coordinates relative to the window which the level is rendered in	ScreenToClient( Hwnd, &ptCursor );	// Compute the pick ray in screen space	v.x =  ( ( ( 2.0f * ptCursor.x ) / ScreenWidth  ) - 1 ) / matProj._11;    v.y = -( ( ( 2.0f * ptCursor.y ) / ScreenHeight ) - 1 ) / matProj._22;    v.z =  1.0f;	// Get the inverse view matrix	m_Graphics->GetDeviceCOM()->GetTransform( D3DTS_VIEW, &matView );	D3DXMatrixInverse( &m, NULL, &matView );    // Transform the screen space pick ray in to 3D coordinates	vPickRayDir.x  = v.x*m._11 + v.y*m._21 + v.z*m._31;    vPickRayDir.y  = v.x*m._12 + v.y*m._22 + v.z*m._32;    vPickRayDir.z  = v.x*m._13 + v.y*m._23 + v.z*m._33;    vPickRayOrig.x = m._41;    vPickRayOrig.y = m._42;    vPickRayOrig.z = m._43;	// Collect the hit information	D3DXIntersect( m_LevelMesh->GetMesh(), &vPickRayOrig, &vPickRayDir, &bHit, &dwFace, &fBarry1, &fBarry2, &fDist, NULL, NULL );	// If a hit is acquired - store the details	if( bHit == TRUE )	{		// Normalise direction		D3DXVECTOR3 Output, Result;		D3DXVec3Normalize( &Output, &vPickRayDir );		// Work out position position		Result = vPickRayOrig +	(fDist*Output);			return Result;	}	else	{		return D3DXVECTOR3(0.0f,0.0f,0.0f);	}}

For a little clarification, in the code above m_LevelMesh->GetMesh() just returns a mesh of type ID3DXMesh*. This is an ordinary mesh that is centred around the origin and not transformed when rendered.

If you want to test this computed pick ray against objects placed arbirtarily in your world then you would need to transform the ray origin and direction by the inverse of the object's world position matrix before doing the intersection test with the object. This moves the ray in to the object's body space for testing.

[Edited by - __Daedalus__ on September 14, 2004 5:34:59 AM]

##### Share on other sites
Quote:
 Original post by EndurionWhat API are you using?

This is the DirectX forum [smile]

##### Share on other sites
Quote:
Original post by Coder
Quote:
 Original post by EndurionWhat API are you using?

This is the DirectX forum [smile]

Erm... yeah... (slaps myself with a spork)

##### Share on other sites
__Daedalus__,

the code you posted assumes the camera is always pointing in 0,0,1 direction right ?

How do I correctly convert the pixel coordinates if the camera is pointing in some random direction ?

Edit: I actually have another question. Since I am setting the persepctive angle to 45 degrees, why cant I simply figure out the ray by hand ? Shouldnt be too hard to figure out the correct angles at every point (if it's linear). Is it linear ? Is it a correct method to implement picking ?

According to my calculations, if the perspetive is 45 degrees horizontally, it should be 33.69 degrees vertically at 640x480. So at the outer edge, if I rotate the ray by 33.69 in the x axis and 45 degrees in the z axis won't it be pependicular to the screen ? Then I can find any point(s) in between.

Thanks,

Nitzan

[Edited by - nitzan on September 14, 2004 5:47:51 PM]

##### Share on other sites
Quote:
 Original post by nitzan__Daedalus__, the code you posted assumes the camera is always pointing in 0,0,1 direction right ? How do I correctly convert the pixel coordinates if the camera is pointing in some random direction ? Edit: I actually have another question. Since I am setting the persepctive angle to 45 degrees, why cant I simply figure out the ray by hand ? Shouldnt be too hard to figure out the correct angles at every point (if it's linear). Is it linear ? Is it a correct method to implement picking ? Thanks, Nitzan
The camera is pointing in any random direction. The camera's arbitrary translation, and rotation values are stored in the view matrix which you retrieve each time in the above function by calling Device->GetTransform( D3DTS_VIEW, &matView ).

Doing these calculations, the pick ray origin is the same position as the the camera (e.g. view matrix) in the world. And the ray direction is the screen space pick ray (a 3D vector built using your mouse coordinates and the projection matrix) transformed in relation to the (inverse) view matrix.

##### Share on other sites
So I implemented the code you posted Daedalus, up to D3DXIntersect. It seems to work fine but I dont understand something. vPickRayOrig always ends up being the camera location, meaning the ray seems to eminate from the center of the screen instead of where I actaully clicked. Also vPickRayDir is a 2 dimensional vector (its Z value is always zero) pointing from the center of the screen toward the point of click. So what I am guessing is that D3DXIntersect does some more work. Any idea how to get the exact point of click and the correct direction the pick ray is traveling at (I am using my own intersection routine with my own model format) ?

Once again:
vPickRayDir seems to be a 2d vector originating at the center of the screen and pointing toward the direction of the mouse click.
vPickRayOrig is always the camera location, meaning the center of the screen.

Thanks!

Nitzan

##### Share on other sites
that is right nitzan. the origin of the ray should be from the cameras location. I had the same problems recently with picking so i know what its like. i came up with a simple analogy to help understand it.

Look out of a window. the glass in the window is your screen, your eye is the camera and everything past the glass is you world. your finger acts as your mouse cursor.

now, choose something you want to pick. (preferable towards the left edge of the window to highlight my point better) say a tree/fencepost/car etc.

now, put your finger on the glass where you see the object you chose. now, your finger should be fully or partly obscuring the object. Now keep you finger there, and move your head to looking at your finger at a right angle to the glass. you will see that you finger (read cursor) is only about 50-100 pixels away from the screen edge.

but, the object you picked is far past the screens area to the left. (lets say, in screen coordinates (ie, relative to the glass/screen) -1000 pixels)

this is why we need to do all the ray stuff, because the object isnt really there, its just where we see it.

when your head, finger and object were in line, your basically projecting a ray from your eye (camera) through your finger (cursor).

another analogy, is with the perspective. to mimic a lower perspective, move your eye further from the window, and vice versa with a higher perspective.

hope that helps you understand. but what is your aim for this? are you trying to select (pick) something in your world? if you are then i suggest using 'bounding volumes'. This way you do your 'checking for a hit' with the bounding volume, and not the mesh (as in daedalus's code above). This is more efficient, but once you have determined it 'hit' the bounding volume, you can then go onto checking which part of the mesh it hit. but i suspect you wont be needing to do this.

##### Share on other sites
Check out the directx Pick sample...cut and paste the code...you're done.

Seems like somebody posts the same question every day or two...and I was guilty of it myself once. MS provided a lot of fairly concise samples with DirectX that illustrate a lot of the concepts that people commonly use, and they included a SampleBrowser app to make samples easy to find. For all of the complaining that everybody does (including myself) about MS's documentation, they made up for that by providing public domain code that can often be pasted right into your program. I think their logic was that it's easier step through the code to see what's going on than to read some really confusing tutorial. We can save ourselves a lot of wheel-reinventing by following the breadcrumbs that Uncle Bill left for us.

##### Share on other sites
Quote:
 Original post by danromeoCheck out the directx Pick sample...cut and paste the code...you're done.Seems like somebody posts the same question every day or two...and I was guilty of it myself once. MS provided a lot of fairly concise samples with DirectX that illustrate a lot of the concepts that people commonly use, and they included a SampleBrowser app to make samples easy to find. For all of the complaining that everybody does (including myself) about MS's documentation, they made up for that by providing public domain code that can often be pasted right into your program. I think their logic was that it's easier step through the code to see what's going on than to read some really confusing tutorial. We can save ourselves a lot of wheel-reinventing by following the breadcrumbs that Uncle Bill left for us.

Microsoft have a habit of releasing convoluted and confusing code. There are so many more resources in book or web tutorial form that are simpler to read and easier to understand. nitzan is doing the right thing by trying to understand what is going on in the code. It is not usually a good idea to just copy-paste code without understanding the algorithm.

##### Share on other sites
if you are looking to get a book to help you on this (despite us working on computers/having the internet, books are an excellent learning tool), then try

Introduction to 3D Game Programming with DirectX 9.0
by Frank D. Luna

it has a chapter all about picking, which is why i got it in the first place, but it also has lots of other information on bounding volumes and even HLSL. I'll try and find a link for it somewhere and edit this post.

EDIT: heres the link to it on this site.

http://www.gamedev.net/columns/books/bookdetails.asp?productid=386

##### Share on other sites
I was wondering if you all can help me since the top is picking. I've been messing around with this off and on for the last two weeks, and am really close, I hope, to having it solved but I've run out of ideas.

To give you the context of the problem, the program models the solar system, kind of taking it to scale was a bad idea. So the planets orbit around the sun and the camera is locked at a 45 degree down angle, no rotation, but the user can scroll the camera around. I started with the DX9 SDK example on picking, and then went and bought the book Intro 3D Programming with DX9, and am using the bounding spheres that he recommends. The problem is that the picking hotspots don't seem to rotate with the planets, they stay where the planet starts. The code for the transformations looks like this:
(pseudo code)

GetViewMatrix(view)

CalculatePickingRay(mouseX, mouseY)

mesh = planets_transformations_during_drawing;

takeInverse(InverseMesh, mesh)

takeInverse(InverseView, view)

TransformRay(view)

TransformRay(mesh)

CheckRaySphereInersect(planet.bounding_sphere)

I've played with the order of transformations, multipling them together, not taking inverses, I did find that if I don't take the inverse of the mesh it does rotate but backwards and much faster than the planet, and taking the transpose of each matrix. So what simple error am I making?

Thanks for the help

##### Share on other sites
EDIT:

WHOPS, silly me. I forgot to set the view back to 3d space. I was drawing some 2d stuff above the 3d objects and forgot to reset it when calculating the pick ray.

Thank you for all the help everyone. It works great now.

Nitzan

##### Share on other sites
@ MountainLion2

i presume your talking og the Frank D Luna book?

you need to transform the bounding volumes to the position of your planets. they wont go there on their own! :P

you could either update their position every cycle, or (and this is what i did, dont know if its the best way though) move the bounding spheres to the planets position when your checking for a hit.

i know the book doesnt spell this out very well, it doesnt mention that you have to move the spheres around (which is common sense really)

when you do your test for a hit, set the boundingspheres center to the planets coordinates, and the radius should be a constant depending on the size of the planet.

i know were kinda hijacking somebody elses thread, but i cant help but help! :P

##### Share on other sites
Multiverse,

Thanks for the help, and I apalogize to the others for jumping in on their thread. I thought it'd be a good place to ask my question without having to start another thread on picking :)

I set the bounding spheres center coord to those of my planets, and I asked the computer to be a pal and draw some spheres to show me where the bounding spheres were. The spheres are moving along with the planets, but when I go to click on them, nothing happens :( Do I need to do something to adjust the ray? I am setting the center corrdinates after the bounding sphere has been calculated.

Oh, and yes its the Frank D. Luna book, which is very good and worth getting. But after this class I'm looking forward to going backk t OpenGL.

MountainLion2

##### Share on other sites
Quote:
 Original post by MountainLion2Multiverse,Thanks for the help, and I apalogize to the others for jumping in on their thread. I thought it'd be a good place to ask my question without having to start another thread on picking :)I set the bounding spheres center coord to those of my planets, and I asked the computer to be a pal and draw some spheres to show me where the bounding spheres were. The spheres are moving along with the planets, but when I go to click on them, nothing happens :( Do I need to do something to adjust the ray? I am setting the center corrdinates after the bounding sphere has been calculated. Oh, and yes its the Frank D. Luna book, which is very good and worth getting. But after this class I'm looking forward to going backk t OpenGL.MountainLion2

With reference to my code above, if you have computed a test ray origin and ray direction in world coordinates, all you need to do is transform both vectors by the inverse of the planet you are testing's world transformation matrix. This moves the ray in to the body space of the planet. Then you do the intersection test.

##### Share on other sites

This topic is 4837 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Create an account or sign in to comment

You need to be a member in order to leave a comment

## Create an account

Sign up for a new account in our community. It's easy!

Register a new account

• ### Forum Statistics

• Total Topics
628730
• Total Posts
2984427

• 25
• 11
• 10
• 16
• 14