Jump to content
  • Advertisement
Sign in to follow this  

Correct way of doing Mouse Picking

This topic is 1023 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello, in the past few days i've been trying to implement Mouse Picking on my 3D game, however i am getting some strange results. Below is the code i am using. If anyone know any algorithm which works or can help me figure out why this doesnt work i will appreciate it very much.

Vector4 world = new Vector4( (2 * mouse.X) / viewport.Width -1, -(2 * mouse.Y) / viewport.Height -1, -1, 1
Matrix4 invProj = Matrix4.Invert(proj);
Vector4 eye = Vector4.Transform(world, invProj);
world = new Vector4(eye.X, eye.Y, -1, 1);
Matrix4 invView = Matrix4.Invert(view);
Vector4 invVec = Vector4.Transform(world, invView);
Vector3 raydirection = new Vector3(invVec.X, invVec.Y, invVec.Z).Normalized();

Share this post

Link to post
Share on other sites
If it helps, this is the code I use to compute a ray for picking (Direct3D9)
Ray Ray::compute(const Vec2 &pos, const Vec2 &size, const Matrix &view, const Matrix &proj)
    float w = size.x;
    float h = size.y;

    float sx = pos.x;
    float sy = pos.y;

    Vec3 v;
    v.x = (((2.0f * sx) / w ) - 1) / proj._11;
    v.y = -(((2.0f * sy) / h) - 1) / proj._22;
    v.z =  1.0f;

    Matrix m = inverseMatrix(view);

    Ray ray;

    ray.dir.x = v.x*m._11 + v.y*m._21 + v.z*m._31;
    ray.dir.y = v.x*m._12 + v.y*m._22 + v.z*m._32;
    ray.dir.z = v.x*m._13 + v.y*m._23 + v.z*m._33;
    ray.pos.x = m._41;
    ray.pos.y = m._42;
    ray.pos.z = m._43;

    ray.dir = normalizeVector(ray.dir);

    return ray;
pos is the mouse position on the screen, size is the width/height of the viewport. The ray returned has pos and dir and is in world frame. Matrix is just a typedef for D3DXMATRIX, Vec2 a typedef for D3DXVECTOR2 and Vec3 is a typedef for D3DXVECTOR3. Edited by Aardvajk

Share this post

Link to post
Share on other sites

One thing that looks like a mistake is this:

"-(2 * mouse.Y) / viewport.Height -1"


Should be +1 at the end, or your values will be between -3 and -1 instead of -1 and 1.

Share this post

Link to post
Share on other sites

vec3 DirectionFromScreen(int x, int y, int sw, int sh, float fov, float z_near, float z_far, float aspect)

mat4 mvm = CAM_MODEL * CAM_VIEW;
vec3 dirX, dirY;
	dirX.x = mvm.m[0];
	dirX.y = mvm.m[4];
	dirX.z = mvm.m[8];

	dirY.x =	mvm.m[1];
	dirY.y =	mvm.m[5];
	dirY.z =	mvm.m[9];

	float a = fov / 2.0;
float cotangent = 1.0 / tan( a * imopi );

float ax = z_near / cotangent;

float screen_w = 2.0*ax;

float screen_h = screen_w;// * yratio;

screen_w = screen_w * aspect;

float scr_coord_x = float(x) / float(sw);
float scr_coord_y = float(sh - y) / float(sh);

vec3 dir = FPP_CAM->ReturnFrontVector();

//move to lower left corner
vec3 start_pos = (FPP_CAM->pos + dir * z_near) + (-dirX * (screen_w / 2.0)) + (-dirY * (screen_h/2.0));

vec3 start = start_pos + (dirX * (screen_w * scr_coord_x)) + (dirY * (screen_h * scr_coord_y));

return Normalize( vectorAB(FPP_CAM->pos, start) );

FPP_CAM->pos camera position


fov is fovy actually, so it has to be converted into horizontal fov


since its opengl code int y has to be scrheight - y


CAM_MODEL is always identity


z_far is not used at all

Edited by WiredCat

Share this post

Link to post
Share on other sites

You might also consider using a pick buffer which is a render target where each pixel contains an ID that is the object that covers that pixel. Therefore sampling the pick buffer under the cursor gives you the object the mouse is over. This doesn't solve box selection though, but is a simple way of doing it that doesnt involve alot of projection/unprojection. Note also that its particularly good for picking terrain, where typically you don't want to be doing ray triangle intersection.

Edited by Dave

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!