Yeah, OpenGL picking is not good. If you want to do pixel-perfect picking, it's fairly simple, and the function gluUnProject is your best friend. Basically, make two calls to gluUnProject with your mouse coords: one with a z value of 0.0f, and the other with a z value of 1.0f. The two 3D coords which result from this are the start and end points of a ray that intersects your view frustum at the near and far clip planes exactly where the user clicked. Now you can use this ray to test against your geometry with some simple 3D math. Go with a coarse-grained test first, like testing the ray against geometry bounding boxes. If the ray intersects a bounding box, go ahead and test the polys of that geometry to see if the ray actually hit a poly. Finally, if a poly was hit (and if you really need to), you can determine the exact point of the ray/poly intersection. Google should be able to provide you with the 3D math you'll need if you're not familiar with it and there are also GDNet articles on it.

You know, I think this would be a good topic for an article, if one doesn't exist on here already... I might write one this week.

I have been also looking for this answer since a long time ago, how to implement a simple mouse coord to 3d world coordinate conversion.

here's a link to a method i've used before. Works well even for 2D, unlike the selection buffer.

Quote:Is that the kind of picking i could use in a 3D RTS or should i use the other method described in the red book consisting on drawing with a different color for each unit ( with lighting disabled, a color would represent a unit id ) in the image buffer ( it could be in several passes if there are many units ) and comparing the color of the pixel under the cursor with the unit colors?

It is your call really. The main goal should just be to get it working. Using OpenGL selection buffer requires drawing all "selectable" entities twice, and it does suffer from precision problems and occasional missed hits. Doing the gluUnproject method means you don't need the additional drawing pass and the precision is much better (not bound to depth buffer precision), but you have to do the ray-object intersection manually. So which to choose depends on the complexity of the objects that need to be selectable, and your comfort level with 3D math and OpenGL in general. If you just want to get something working, go with OpenGL selection buffers. Once you find that it limits you, look into more sophisticated methods.

@Ashkan - that would probably make a really good article, you should contact the staff.