I've been trying to implement mouse picking in an RTS game, in DirectX11. Been reading tutorials and got everything working if the camera doesn't rotate. However, as soon as the camera rotates, the point in world space zooms off away from the mouse. The camera is looking down on the scene and the rotation is around the Y axis. This is the code I'm using:


float3 v;
v.x = ( ( ( 2.0f * MousePos.x ) / sWidth ) - 1 ) / Camera->ProjMatrix._11;
v.y = -( ( ( 2.0f * MousePos.y ) / sHeight ) - 1 ) / Camera->ProjMatrix._22;
v.z = 1.0f;

XMMATRIX ViewMatrix = ConvertToXM(Camera->ViewMatrix);
XMMATRIX m = XMMatrixInverse(NULL, ViewMatrix);

// Transform the screen space pick ray into 3D space
float3 rayOrigin,rayDir;
rayDir.x = v.x*m._11 + v.y*m._21 + v.z*m._31;
rayDir.y = v.x*m._12 + v.y*m._22 + v.z*m._32;
rayDir.z = v.x*m._13 + v.y*m._23 + v.z*m._33;
rayOrigin.x = Camera->CameraPosition.x;
rayOrigin.y = Camera->CameraPosition.y;
rayOrigin.z = Camera->CameraPosition.z;

// x and z position when y is 0
float y = rayOrigin.y/rayDir.y;

ProjectedMousePos.x = (-rayDir.x*y + rayOrigin.x);
ProjectedMousePos.z = (rayDir.z*y + rayOrigin.z);
ProjectedMousePos.y = 0;


How are you meant to handle camera rotation when picking?

Right, I tried implementing your sugestions, but it's still not working. The pickRayInWorldSpacePos vector is always 0, 0, 0, shouldn't it be at the camera position?


float3 v;
v.x = ( ( ( 2.0f * MousePos.x ) / sWidth ) - 1 ) / Camera->ProjMatrix._11;
v.y = -( ( ( 2.0f * MousePos.y ) / sHeight ) - 1 ) / Camera->ProjMatrix._22;
v.z = 1.0f;
XMVECTOR vVector = XMVectorSet(v.x, v.y, v.z, 0.0f);

XMMATRIX ViewMatrix = ConvertToXM(Camera->ViewMatrix);
//XMMATRIX m = XMMatrixInverse(NULL, ViewMatrix);

XMVECTOR matInvDeter; //Not actually used but needed for the next function
XMMATRIX pickraytoworldmatrix = XMMatrixInverse(&matInvDeter, ViewMatrix);
XMVECTOR pickRayInWorldSpacePos = XMVector3TransformCoord(XMVectorSet(0, 0, 0, 0), pickraytoworldmatrix );
XMVECTOR pickRayInWorldSpaceDir = XMVector3TransformNormal(vVector, pickraytoworldmatrix );

// x and z position when y is 0
float y = pickRayInWorldSpacePos.m128_f32[1]/pickRayInWorldSpaceDir.m128_f32[1];

ProjectedMousePos.x = (-pickRayInWorldSpaceDir.m128_f32[0]*y + pickRayInWorldSpacePos.m128_f32[0]);
ProjectedMousePos.z = (pickRayInWorldSpaceDir.m128_f32[2]*y + pickRayInWorldSpacePos.m128_f32[2]);
ProjectedMousePos.y = 0;


This is some pseudo-code to (hopefully) help you illustrate Jason's journal entry. I hope I didn't make any mistakes. I did, however, assume your camera has a means of returning its own position in world-space instead of multiplying the origin point with the camera's view matrix in order to get it. Not sure why you decided to go that way?

[source lang="cpp"]
float3 start_point, end_point;

// begin the ray right where the camera is at (in world space)
start_point.set ( camera.get_position ( ) );

// first, set the end point to the screen space position on the far plane
end_point.set_x ( ( mouse.get_x ( ) / window.get_width ( ) ) * 2.0f - 1.0f );
end_point.set_y ( ( mouse.get_x ( ) / window.get_width ( ) ) * 2.0f - 1.0f );
end_point.set_z ( 1.0f );

// take the inverse of the view * projection matrix
float3x3 inverse_view_projection_matrix = ( float3x3 ) ( camera.get_view_matrix ( ) * camera.get_projection_matrix );
inverse_view_projection_matrix = math.inverse_matrix ( &inverse_view_projection_matrix, 0.0f );

// multiply the screen-space position on the far plane with the inverse view projection matrix to bring the end point from screen to world space
end_point *= inverse_view_projection_matrix;

// now we only have to create our ray (vector) from the two points
float3 ray ( end_point - start_point );
[/source]

I hope this helps you (and, in turn, me) to fully understand ray picking! Please note that I am not an expert on the topic yet, so don't take my pseudo-code as tested and working. I was just going through Jason's write-up myself!

I don't think I'm doing this right .



XMVECTOR start, end;

// begin the ray right where the camera is at (in world space)
start = XMVectorSet(Camera->CameraPosition.x, Camera->CameraPosition.y, Camera->CameraPosition.z, 1.0f);

// set the end point to the screen space position on the far plane
end = XMVectorSet((( MousePos.x / sWidth) * 2.0f - 1.0f ), (( MousePos.y / sHeight) * 2.0f - 1.0f ), 1.0f, 1.0f);

// take the inverse of the view * projection matrix
XMMATRIX view = ConvertToXM(Camera->ViewMatrix);
XMMATRIX proj = ConvertToXM(Camera->ProjMatrix);
XMMATRIX inverted = view * proj;
inverted = XMMatrixInverse(NULL, inverted);

// multiply the screen-space position on the far plane with the inverse view projection matrix to bring the end point from screen to world space
end = XMVector4Transform(end, inverted);

// now we only have to create our ray (vector) from the two points
XMVECTOR ray = end - start;

// x and z position when y is 0
float y = start.m128_f32[1]/ray.m128_f32[1];
ProjectedMousePos.x = ray.m128_f32[0]*y + start.m128_f32[0];
ProjectedMousePos.z = ray.m128_f32[2]*y + start.m128_f32[2];
ProjectedMousePos.y = 0;