I'm using an ortho left hand setup and have a few questions. An answer to any of these would give me something to work with. I have a camera in worldspace (A), and a desired second point in worldspace (B). How would I get a ray direction vector from A->B? How could I generate a 3d coordinate in worldspace from where the mouse was clicked in screenspace? I have a viewport (512x512) where the mouse will be clicked on (a picturebox) to select objects. How would I get a vector in worldspace for this click, and a direction vector? It should account for camera position and camera target in the projection matrix. The final implementation will use boxboundprobe to check for a collision with an object. I've done this multiple times with an fov setup all with less than satisfactory results.

Quote:Original post by iosys
I have a camera in worldspace (A), and a desired second point in worldspace (B). How would I get a ray direction vector from A->B?

Well if both points are in the same space, a simple (B - A) should suffice, normalize it if you require a unit vector...

Quote:Original post by iosys
How could I generate a 3d coordinate in worldspace from where the mouse was clicked in screenspace?

I would make use of D3DXVec3Unproject( ). Provided you have access to all the information it requires, it'll do the hard work for you. Just be a little careful if you're using a pure device as you won't be able to use any Get*( ) calls to retrieve the data for the params.

Quote:Original post by iosys
I have a viewport (512x512) where the mouse will be clicked on (a picturebox) to select objects. How would I get a vector in worldspace for this click, and a direction vector?

To be completely honest, I'm not sure what the aforementioned function requires as it's input coordinates. The documentation isn't clear... your mouse click will give you a 2D coordinte (which I'd shove into the .X and .Y fields of the D3DXVECTOR3) but that still leaves the .Z field... I'd try either a 0.0f or 1.0f value for these - probably the latter!

hth
Jack

Jack, good to hear from you. I decided to drop VB6 completely and work in .NET, so naturally I went for your DX9 PDF which helped me get back up and running.

Would the direction vector be the 31,32,33 from the view matrix since its an ortho projection?

Doing this:

Dev.Transform.Projection = Matrix.OrthoLH(frm.picVP.Width, frm.picVP.Height, -10000, 10000)Dev.Transform.View = Matrix.LookAtLH(New Vector3(0, 0, -300), New Vector3(0, 0, 0), New Vector3(0, 1, 0))vec = Vector3.Unproject(New Vector3(MouseX, mouseY, 0), vp, Dev.Transform.Projection, Dev.Transform.View, Dev.Transform.World)dir = New Vector3(Dev.Transform.View.M31, Dev.Transform.View.M32, Dev.Transform.View.M33)Geometry.BoxBoundProbe(New Vector3(0, 0, 0), New Vector3(64, 64, 64), vec, dir)

mousex and mousey are the mouse coordinates in screen space. boxboundprobe returns true until I move the camera up (change the y for the camera position vector in the view matrix).

[Edited by - iosys on June 13, 2005 5:49:33 PM]

Looks like I got it working. I figured if I unprojected two times, one for the near plane and one for the far plane, I could then subtract near from far to get the direction. I hope this remains the solution in the long run!

vNear = Vector3.Unproject(New Vector3(MouseX, mouseY, 0), vp, Dev.Transform.Projection, Dev.Transform.View, Dev.Transform.World)vFar = Vector3.Unproject(New Vector3(MouseX, mouseY, 1), vp, Dev.Transform.Projection, Dev.Transform.View, Dev.Transform.World)dir = Vector3.Subtract(vFar, vNear)

This is a post you can look at. Helped me out.
http://www.gamedev.net/community/forums/topic.asp?topic_id=322931

Yes, "paic" posted the same method in that other thread. I believe the 0 and 1 used for z in the near and far vectors correspond to the viewport object nearZ and farZ. This might be why I could never get raypick in my fov setups since I wasn't matching the near/far of the viewport and vector z's. I have implemented boxboundprobes for each object in the scene and picking objects works perfectly.