I got this code to compute the pickray and then transform it into objectspace:
void createPickray(int mouseX, int mouseY){
float pointX, pointY;
windowW = 1024 - 16; //1008;
windowH = 768 - 38; //730;
// Move the mouse cursor coordinates into the -1 to +1 range.
pointX = ((2.0f * (float)mouseX) / (float)windowW) - 1.0f;
pointY = (((2.0f * (float)mouseY) / (float)windowH) - 1.0f) * -1.0f;
// Adjust the points using the projection matrix to account for the aspect ratio of the viewport.
projectionMatrix = cam1->getcamprojm();
pointX = pointX / projectionMatrix._11;
pointY = pointY / projectionMatrix._22;
// Get the inverse of the view matrix.
viewMatrix = cam1->getcamviewm();
D3DXMatrixInverse(&inverseViewMatrix, NULL, &viewMatrix);
// Calculate the direction of the picking ray in view space.
direction.x = (pointX * inverseViewMatrix._11) + (pointY * inverseViewMatrix._21) + inverseViewMatrix._31;
direction.y = (pointX * inverseViewMatrix._12) + (pointY * inverseViewMatrix._22) + inverseViewMatrix._32;
direction.z = (pointX * inverseViewMatrix._13) + (pointY * inverseViewMatrix._23) + inverseViewMatrix._33;
// Get the origin of the picking ray which is the position of the camera.
origin = cam1->getpos();
}
// To objectspace:
void Pick(D3DXVECTOR3 origin, D3DXVECTOR3 direction){
D3DXMATRIX inverseWorldMatrix;
D3DXVECTOR3 rayOrigin, rayDirection;
// Now get the inverse of the translated world matrix.
D3DXMatrixInverse(&inverseWorldMatrix, NULL, &World);
// Now transform the ray origin and the ray direction from view space to world space.
D3DXVec3TransformCoord(&rayOrigin, &origin, &inverseWorldMatrix);
D3DXVec3TransformNormal(&rayDirection, &direction, &inverseWorldMatrix);
// Normalize the ray direction.
D3DXVec3Normalize(&rayDirection, &rayDirection);
}
When using this on an object with default worldmatrix, everything runs just fine. But in any different worldmatrix, it completly messes up.
After some further investigation: The Scaling part runs fine, only translation and rotation are not applied, even when I use only them.
Can someone tell me whats wrong here?
