So far I've implemented Ray picking for my 3D terrain engine, I can click on a unit or the map and it will tell me were I clicked. It does it by shooting a ray from the camera to the mouse and checking if it hits a unit (a sphere around the unit) or the terrain. But how do I make it select multiple units if I drag the mouse and make a 2D square on the screen? Thanks.

Hmm.. deleted my first post because I figured out a better way to do it. How about this:

Variables:
vLowerleft, vUpperright - 2 rays that make your drag frustrum.
vToObject - the object you are checking, do this for every object.
vForward, vRight, vUp - orientation of the camera/player.

You want to select the object if vToObject is left of vUpperright, below vUpperright, right of vLowerleft, and above vLowerleft. And left, right, above, and below are arbitrary based on your camera orientation.

For each object:
1) Get the vector to it (vToObject).
2) If vToObject dot vForward < 0, trivially reject the object; its behind you.
3) Take each vector (vUpperright, vLowerleft, vToObject) and dot it with vUp. Lets call the results high, low, and object, respectively. Remeber, these are floats, not vectors.
4) If object is between low and high, its in the drag frustrum. If not, skip the object.
5 & 6) Do the same for vRight. Dot each vector with vRight and make sure the object is between the two.

If you need more help visualizing whats going on, mathworld.com has some solid visuals of the dot product.

Thanks, I want to try that but I have one question.
Lets say I have:
gluLookAt(10,40,20, 0,-0.2,-1, 0,1,0);

How do I get the vUp, vRight, and vForward vectors?
Thanks.

I would think it woudl be trivial to just generate 4 "clip planes" from that box with normals facing inwards, and do a plane/bounding box test

cheers
-Dan

Quote:Original post by Ademan555
I would think it woudl be trivial to just generate 4 "clip planes" from that box with normals facing inwards, and do a plane/bounding box test

cheers
-Dan

Its not quite that easy. A rectangle on your screen gets mapped to a view frustrum in the 3D world, not a bounding box. Basically, instead of having straight edges, its more of 3D trapezoid. But testing the 4 clip planes is exactly what the algorithm is doing.

To get the forward, right, and up vectors from your opengl call, your parameters include the eye point, look at point, and up vector.
Forward is the look at point minus eye point.
Up is given.
Right is Up Cross Forward (or Forward X Up if you are using left-handed coord. system, I think thats what openGL uses).

There is a little problem with my algorithm. You don't know how long vUpperright and vLowerleft rays are in the 3D world. Make sure you normalize vUpperright, vLowerleft and vToObject before you dot them to your camera directions.

blaze - I tried you algorithm and it worked! Thanks!