I am having some problems with ray picking in DirectX. I am using it for a 3d map editor; the objects to be selected are not meshes, but simple vertex-defined quads with textures. I cannot fully emulate the pick example in the SDK; while it was useful for demonstrating the principles underlying the process, it is designed for meshes, and uses the D3DXIntersect function which takes a mesh as an argument. I have picked one of the quads as a test object and am trying to get the program to pick that object when the mouse is over it. I am using a plane intersection check, but the intersection vector is always wrong. I am coding it like this. Apologies for the slightly lengthy nature, but might as well make sure all the bases are covered: The problem is very simple: vecIntersect, even though the plane is being drawn at the same Z coordinate of the test object, is always way, way off from the object's actual world coordinates (g_x, g_y, and g_z). I haven't been able to notice any pattern in the discrepancy either, it's just way off. I can't for the life of me figure why. A vector is derived from the mouse position, another vector down the z axis with the same x y coordinates, a plane is drawn at the z coordinate of the test object I'm trying to check, and it checks for an intersection. But so far no dice, which is perplexing because I think the way I've done it is pretty much identical to the plane/intersection method for picking that I've seen in other threads and tutorials. It seems that, for some reason, the mouse coordinates are just not gelling with the world coordinates of the object; it's as if the mouse and the object I'm trying to check are in two different world transforms, but I have checked and made absolutely certain that the world transform is constant. I'm follow the method described by Zaei (I'm assuming from the signature) in the last post on this thread: http://www.gamedev.net/community/forums/topic.asp?topic_id=48728 Any ideas or suggestions? I am probably missing something very obvious and silly. Thanks, sayfadeen [Edited by - sayfadeen on September 21, 2004 4:10:08 PM]

One obvious thing I have noticed with this is that x,y coordinates of the untransformed mouse position do not match up with the x,y coordinates of the object which is under it. Even with a perspective view, shouldn't the coordinates match up (I'm just thinking about it visually here. If I'm standing here and facing directly at an object at a distance and we are aligned horizontally, doesn't this by definition mean that my "x coordinate" and its "x coordinate" should be the same?). I have also noticed that the untransformed x,y of vecMouse and vecFar are different from each other, even though the original x,y is the same (ptCursor.x, ptCursor.y). Is this supposed to happen because of the projection/perspective?. If this is supposed to happen, then clearly I'm missing something about how projection works (I'm going to RTFM again anyways). But if this is not supposed to happen, then what to do to scale the untransformed mouse coordinates in such a way that when the mouse cursor is over an object, the mouse x,y will match the object's x,y?

[Edited by - sayfadeen on September 21, 2004 3:01:30 PM]

Anybody? :)

Yeah, you should keep the projected x,y and z coords as you do.
What bothers me, is:

Quote:
D3DXVec3Unproject(&vecMouse, &vecMouse, &vpScreen, &matProj,
&matView, &matWorld);

vecMouse.z = 0.99f;

D3DXVec3Unproject(&vecFar, &vecMouse, &vpScreen, &matProj,
&matView, &matWorld);

You are overwriting vecMouse in the first call, then you are using it again to compute vecFar. Reverse order of those calls and it shold be "much better" ;)

Also, your plane is perpendicular to XY plane in world coordinates, not in screen coords. What you could do to construct the plane you want, is:

- take a vector(0.0f, 0.0f, 0.5f) in the meaning of screen coords.
- unproject it, then it would represent the "center"(or so) of the original camera frustum.
- substract it from your camera's position, then you will get your camera's direction(or you can just take your camera's direction from somewhere else, that would be much simplier)
- normalize the direction and create the plane from that normal and (g_x, g_y, g_z).

I don't know if I understood your problem right, but hopes it is of some help.
/def

Yeah, that helps, and thank you very much for your response. I'm still a bit unclear about a couple points, being quite a newbie at this.

To get the camera position, should I just take the ._41, ._42, ._43 values from the view matrix (or possibly the inverted matrix)? And would I just use D3DXVec3Normalize on the camera position? I've tried it these different ways but no success, so I'm probably doing it quite wrong.

[Edited by - sayfadeen on September 22, 2004 4:47:21 PM]

Quote:
should I just take the ._41, ._42, ._43 values from the view matrix

Just out of curiosity, where did you get your view matrix from? I always construct it from camera's position and it's rotation(which gives me its look direction).
Anyway, yes you inverse the matrix first, so as you move in a direction, it's like all your objects were moving in the opposite. If you rotate left, it's like your world rotated right and you stood still the entire time(it is not _that_ simple, but taking translations and rotations independently, it can be done by hand like that).

Then you could take those values(fourth column). This would be _position_, so you didn't want to normalize this.
Another story with direction. You could also accuire it from inversed view matrix(it's third column, as it determines transformation of your Z axis to your new Z axis, which is, in the end, just where camera looks). You don's need to normalize it, as it is normalized already(if it wasn't your view in z-dir would be somewhat streched).
But much better(and cleaner) it would be if you tracked the way your view matrix is _born_, then you could have all the needed values directly.

Try it out.
/def

PS: I never, ever had to use inversion of a matrix in any of my DX3D programs. Not that I _avoided_ it or so. It is just not necesarry, IMO.

D3DXIntersectTri

You just give it a triangle, not a mesh. The rest is up to you but I'm sure you can figure it out from here.

Well they both worked! I'm surprised there's two so different ways of going the same thing.

Excellent work, I really appreciate the help. I was starting to despair for a minute there.

Kind regards,
Sayfadeen