One method someone else will suggest is to render your scene once where every object has a unique color and then read the color underneath your mouse cursor to get the color of the object that is there (and then you know what object).

That is a good technique and is supposed to be pretty fast, fast enough to do every frame from what i understand.

What i personally like doing though is casting a ray into the scene and seeing what it hits.

Plain vanilla ray casts against all objects is REALLY CPU intensive though!

So to make it faster...

#1 - organize your objects into some spatial partitioning system, like a grid, a quad tree, an octtree, something like that. When you cast your ray, use the spacial system to severely cut down on the number of objects to test

#2 - instead of testing the ray against the model of each object straight away, test first against a bounding sphere or bounding box, and if that succeeds, then test against the model (or maybe bounding box or sphere is good enough for you and you can stop there).

Anyhow, thats how i do it.

Another option (instead of color coding) would be to use occlusion queries with a scissor rectangle of one pixel (at the mouse position).

The nice thing about occlusion queries is that they count the number of fragments that passed the depth test -- note the wording. This means you don't really need to draw anything, color writes can be disabled so you can directly "draw" into the same scene. It also means that you can use the stencil buffer (stencil test before depth!) to specifically disable the hit test where you don't want it, if necessary. For example, all GUI elements could set stencil so if you click on the screen where there is a GUI object, it will never return a hit test for an object in the scene. Or, anything else you can imagine.

Also, occlusion queries don't require you to create an extra framebuffer, read back pixel data and do some hackery with some obscure data in memory. The API is very clear and easy, and you get back an easy number that tells you what you need.

The downside is that you will possibly have to iterate over several dozen or a hundred query results, but that shouldn't be much of a problem. Also, you can easily sort out objects that cannot possibly be hit beforehand (using frustum techniques or similar).