• 12
• 12
• 9
• 10
• 13

# Picking with a "weird" coordinate system

This topic is 3567 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I'm trying to implement picking using the method described in the DXSDK sample, and I'm running into some problems. I use the 3ds max coordinate system (right handed with z-up) which may be one of the reasons why it's not working. But nothing I try has any results (it never reports a "pick"). Here's my code:
const Matrix& projection = camera->GetProjectionMatrix();

Vector3 pickRay;
pickRay.x = ((2.0f * mouseX) / camera->GetViewportWidth() - 1.0f) /
projection(0, 0);
pickRay.y = -((2.0f * mouseY) / camera->GetViewportHeight() - 1.0f) /
projection(1, 1);
pickRay.z = 1.0f;

Matrix viewInverse = camera->GetViewMatrix();
viewInverse.Invert();

m_direction.x = pickRay.x * viewInverse(0, 0) +
pickRay.y * viewInverse(1, 0) + pickRay.z * viewInverse(2, 0);
m_direction.y = pickRay.x * viewInverse(0, 1) +
pickRay.y * viewInverse(1, 1) + pickRay.z * viewInverse(2, 1);
m_direction.z = pickRay.x * viewInverse(0, 2) +
pickRay.y * viewInverse(1, 2) + pickRay.z * viewInverse(2, 2);
m_direction.Normalize();

m_origin.x = viewInverse(3, 0);
m_origin.y = viewInverse(3, 1);
m_origin.z = viewInverse(3, 2);

m_origin += m_direction * camera->GetNearClippingPlane();


The matrices I use are of the same layout as the ones in DirectX (row number first). If I switch x and z on the direction and/or origin vectors, I get incorrect picking results (reports a "pick" when nowhere near an object). I guess all I need is a hint as to where I should look. Thanks a lot in advance!