Jump to content
  • Advertisement
Sign in to follow this  

Mouse ray/picking trouble

This topic is 2799 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts


I am using Sony's Vectormath from Bullet Physics download and DirectX 11. I'm trying to add mouse picking, but I'm having some troubles creating ray. Maybe someone could help me out? Here's my code:

const Matrix4 &Proj = gCamera.GetProjMatrix();
const Matrix4 &ViewInv = inverse(gCamera.GetViewMatrix());
const Vector3 MouseRay(
(((2.0f * MousePos[0]) / Viewports[0].Width) - 1) / Proj[0][0],
-(((2.0f * MousePos[1]) / Viewports[0].Height) - 1) / Proj[1][1],
const Vector3 &Direction = normalize(Vector3(
dot(ViewInv[0].getXYZ(), MouseRay),
dot(ViewInv[1].getXYZ(), MouseRay),
dot(ViewInv[2].getXYZ(), MouseRay)
const Vector3 &Start = ViewInv[3].getXYZ();

Start seems to be correct, however direction is not correct. Direction[1] is positive (y axis), but my camera is facing down, so it should have been negative.

Thank you in advance.

Share this post

Link to post
Share on other sites
Tried to untransform screen pixel using inverse Proj*View matrix, but it's not working either. Direction is more-or-less correct; if my mouse is at the center of screen it works correctly, however when I move mouse away ray's direction doesn't change fast enough, that is my mouse is near the edge of window, but ray gets casted just a little bit away from screen's center. Code:

const Vector4 MouseRay(
MousePos[0] - Viewports[0].Width / 2,
Viewports[0].Height / 2 - MousePos[1],
const Matrix4 &ProjViewInv = inverse(gCamera.GetProjViewMatrix());
const Vector3 &Direction = normalize((ProjViewInv * MouseRay).getXYZ());

Anyone knows how to get it working correctly?

Thank you in advance.

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!