mouseInWorld.x = ((((2.0f*mousex)/(float)viewport)-1.0f)/perspectiveMatrix._11);
mouseInWorld.y = ((((2.0f*((float)viewport-mousey))/(float)viewport)-1.0f)/perspectiveMatrix._22);
mouseInWorld.z = -1.0f;
rayPosition.x = IViewInverse._41;
rayPosition.y = IViewInverse._42;
rayPosition.z = IViewInverse._43;
Edited by BornToCode, 07 September 2012 - 10:52 PM.
I guess the topic says it all. The problem that i am having is that i am creating a 3d editor, where you can switch from perspective to ortho or ortho to perspective however you see fit. Now the problem that i am having is that my collision code for the RayToOOBB works fine as long as i have a perspective matrix set up. As soon as i change the matrix to Ortho. The algorithm does not work, How does tool like 3d max handles that. I kind of figure that the same algorithm would work in ortho mode. If anyone have any idea on how to solve that problem i would be greatly appreciated. Here is the piece of code that is converting my mouse from 2d to 3d.