I guess the topic says it all. The problem that i am having is that i am creating a 3d editor, where you can switch from perspective to ortho or ortho to perspective however you see fit. Now the problem that i am having is that my collision code for the RayToOOBB works fine as long as i have a perspective matrix set up. As soon as i change the matrix to Ortho. The algorithm does not work, How does tool like 3d max handles that. I kind of figure that the same algorithm would work in ortho mode. If anyone have any idea on how to solve that problem i would be greatly appreciated. Here is the piece of code that is converting my mouse from 2d to 3d.

GLGXVECTOR3 mouseInWorld;
GLGXVECTOR3 rayPosition;
GLGXVECTOR3 rayDirection;

GLint viewport[4];glGetIntegerv(GL_VIEWPORT,viewport);

mouseInWorld.x = ((((2.0f*mousex)/(float)viewport[2])-1.0f)/perspectiveMatrix._11);
mouseInWorld.y = ((((2.0f*((float)viewport[3]-mousey))/(float)viewport[3])-1.0f)/perspectiveMatrix._22);
mouseInWorld.z = -1.0f;

GLGXMATRIX IViewInverse;
GLGXMatrixInverse(&IViewInverse,NULL,&cameraMatrix);
GLGXVec3TransformNormal(&rayDirection,&mouseInWorld,&IViewInverse);
rayPosition.x = IViewInverse._41;
rayPosition.y = IViewInverse._42;
rayPosition.z = IViewInverse._43;

So I need to convert my ray position to screen space correct.

It's not a matter of converting ray position to screen space. It's a matter of generating a ray for every screen position. For perspective, imagine the rays as streams of water squirting from a single nozzle and fanning out in a cone shape. The rays start from a common point and diverge. For orthographic, imagine the rays as coming from one of those fancy shower heads with hundreds of little nozzles all in a neat array. All the streams are parallel to each other, and the resulting spray shape is more cylindrical than conic. The rays never start from the same point, and in fact never cross each other (barring gravity, of course). It's the same idea with an ortho matrix. The idea of a single ray start position really makes no sense for an ortho projection. If you move your camera straight back, things on the screen remain exactly the same; there is no dwindling in size with distance, as there would be with rays that diverge from a single point.

From what I understood from your post is that I need to figure out two different points in screen space. One at the camera position And one at the object position. If that is bot the case can you show me an example on how you would do it. Or do you mean I need to test multiple rays per object. If that is the case how do I compute those rays then.

Thanks.

Thanks alot. I got it working. Lauris Kaplinski i tried to give you a thumbs up but it was not letting me boost your reputation.