# Correct way of doing Mouse Picking

This topic is 1023 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hello, in the past few days i've been trying to implement Mouse Picking on my 3D game, however i am getting some strange results. Below is the code i am using. If anyone know any algorithm which works or can help me figure out why this doesnt work i will appreciate it very much.

Vector4 world = new Vector4( (2 * mouse.X) / viewport.Width -1, -(2 * mouse.Y) / viewport.Height -1, -1, 1

Matrix4 invProj = Matrix4.Invert(proj);
Vector4 eye = Vector4.Transform(world, invProj);

world = new Vector4(eye.X, eye.Y, -1, 1);

Matrix4 invView = Matrix4.Invert(view);

Vector4 invVec = Vector4.Transform(world, invView);

Vector3 raydirection = new Vector3(invVec.X, invVec.Y, invVec.Z).Normalized();


##### Share on other sites
If it helps, this is the code I use to compute a ray for picking (Direct3D9)

Ray Ray::compute(const Vec2 &pos, const Vec2 &size, const Matrix &view, const Matrix &proj)
{
float w = size.x;
float h = size.y;

float sx = pos.x;
float sy = pos.y;

Vec3 v;
v.x = (((2.0f * sx) / w ) - 1) / proj._11;
v.y = -(((2.0f * sy) / h) - 1) / proj._22;
v.z =  1.0f;

Matrix m = inverseMatrix(view);

Ray ray;

ray.dir.x = v.x*m._11 + v.y*m._21 + v.z*m._31;
ray.dir.y = v.x*m._12 + v.y*m._22 + v.z*m._32;
ray.dir.z = v.x*m._13 + v.y*m._23 + v.z*m._33;
ray.pos.x = m._41;
ray.pos.y = m._42;
ray.pos.z = m._43;

ray.dir = normalizeVector(ray.dir);

return ray;
}

pos is the mouse position on the screen, size is the width/height of the viewport. The ray returned has pos and dir and is in world frame. Matrix is just a typedef for D3DXMATRIX, Vec2 a typedef for D3DXVECTOR2 and Vec3 is a typedef for D3DXVECTOR3. Edited by Aardvajk

##### Share on other sites

One thing that looks like a mistake is this:

"-(2 * mouse.Y) / viewport.Height -1"

Should be +1 at the end, or your values will be between -3 and -1 instead of -1 and 1.

##### Share on other sites
Here's a good way of doing picking in directx 11:

http://www.braynzarsoft.net/viewtutorial/q16390-24-picking

I used this originally but I'vemodified mine slightly to not use d3dx. It works well and is straightforward :)

##### Share on other sites

vec3 DirectionFromScreen(int x, int y, int sw, int sh, float fov, float z_near, float z_far, float aspect)
{

mat4 mvm = CAM_MODEL * CAM_VIEW;
mvm.Transpose();
vec3 dirX, dirY;
dirX.x = mvm.m[0];
dirX.y = mvm.m[4];
dirX.z = mvm.m[8];

dirY.x =	mvm.m[1];
dirY.y =	mvm.m[5];
dirY.z =	mvm.m[9];

float a = fov / 2.0;
float cotangent = 1.0 / tan( a * imopi );

float ax = z_near / cotangent;

float screen_w = 2.0*ax;

float screen_h = screen_w;// * yratio;

screen_w = screen_w * aspect;

float scr_coord_x = float(x) / float(sw);
float scr_coord_y = float(sh - y) / float(sh);

vec3 dir = FPP_CAM->ReturnFrontVector();

//move to lower left corner
vec3 start_pos = (FPP_CAM->pos + dir * z_near) + (-dirX * (screen_w / 2.0)) + (-dirY * (screen_h/2.0));

vec3 start = start_pos + (dirX * (screen_w * scr_coord_x)) + (dirY * (screen_h * scr_coord_y));

return Normalize( vectorAB(FPP_CAM->pos, start) );
}


FPP_CAM->pos camera position

fov is fovy actually, so it has to be converted into horizontal fov

since its opengl code int y has to be scrheight - y

CAM_MODEL is always identity

z_far is not used at all

Edited by WiredCat

##### Share on other sites

You might also consider using a pick buffer which is a render target where each pixel contains an ID that is the object that covers that pixel. Therefore sampling the pick buffer under the cursor gives you the object the mouse is over. This doesn't solve box selection though, but is a simple way of doing it that doesnt involve alot of projection/unprojection. Note also that its particularly good for picking terrain, where typically you don't want to be doing ray triangle intersection.

Edited by Dave

1. 1
2. 2
3. 3
Rutin
19
4. 4
khawk
14
5. 5
frob
12

• 9
• 11
• 11
• 23
• 12
• ### Forum Statistics

• Total Topics
633657
• Total Posts
3013203
×