I am attempting to implement point selection by using the gluUnProject function. The problem I am having is that the z object coordinate it returns is in the range [-1, -512], the near and far planes of my perspective projection. I have been unable to figure out how to convert this value into model coordinates, which I need to do in order to find the point closest to the mouse click since the coordinates of all my points are in model coordinates. Any suggestions?

Maybe you already know this, but a 2d position in window coordinates can't be uniquely mapped to a 3d position in world space. Intuitively this makes sense because an infinite number of 3d points project to the same point onscreen.

When you call gluUnproject() you submit a window z value that is in the current depth range, typically 0 to 1 unless you've changed it. This value maps to the range near to far in camera space. The window z value is basically a value that you have to 'make up' when you call gluUnproject(). What value are you submitting for this?

In any case, for geometric picking the idea is not to get a point in 3d space and test objects to see if they're near it, but to get a ray that passes through the window coordinate and test it for intersection with your objects. You could also use feedback mode and a pick matrix if you prefer, but that's a different subject.

I don't know if that helped at all, but I'll be glad to provide any further info you might need.

I have tried two methods to find the closest point:
(1) Find the depth value from the Z-buffer (pt is the mouse position):

int winX, winY;
winX = pt.x;
winY = viewport[3] - pt.y;

float winZ;
glReadPixels(winX, winY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ);

gluUnProject(winX, winY, winZ, modelview, projection, viewport, &objX, &objY, &objZ);

(2) Ray Tracing (winX and winY as above):
gluUnProject(winX, winY, 0, modelview, projection, viewport, &objX1, &objY1, &objZ1);
gluUnProject(winX, winY, 1, modelview, projection, viewport, &objX2, &objY2, &objZ2);

The problem I'm having is that I do not know how to interpret the z-value output by gluUnProject. The z value it outputs is not a modelview coordinate, it is relative to my prespective projection and I would like to convert it to modelview coordinates.

i.e.) (for the ray tracing case)
I set up my prespective function like this:
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPerspective(60, aspect, 1, 512);

and the results I get for ray tracing (an arbitrary clicked point) are:
(with z=0): (0.0323, -0.3216, -1.0000)
(with z=1): (16.537, -164.684, -512.002)

but the z-values of all the vertices I plot are in the range [9.054, 117.334]
so I'm not sure how to do ray tracing if my z-values are not compatible

Thanks for your help.

Quick question. Currently, is your modelview matrix always identity?

If u are not particular about using gluUnproject , there is a function called gluPickMatrix that does all the picking function for u.

Quote:there is a function called gluPickMatrix that does all the picking function for u

Yeah, I mentioned that earlier. Still, it doesn't really 'do all the picking for you'. You still have to set up and manage feedback buffers, and then render your geometry a second time in feedback mode. But it is an alternative.

My modelview matrix is not always the identity.

My program will generally have multiple shapes with 1000s of vertices on the screen at one time. Since I want to find the closest vertex to the clicked point, I don't think selection mode would be ideal. From what I've read, I would need to label every vertex to find the ones around the clicked point, and there is a limit on the number of names that can be used.

I don't have a definite answer, but I will say that the results you were getting from unprojecting at winz = 0 and 1 looked reasonable. It also looked to me like in that example at least the modelview was identity. In that circumstance, the returned points should be at the near at far planes. In other words, it looks like it's working to me; you should be able to take those two points and create a selection ray or line segment from them.

If you're trying to select individual vertices, here's an idea of how you might proceed (you may have already thought of this). Find all the vertices within a given (squared) distance of your picking ray. Then do a partial transformation into camera space to find which of these is closest to the camera, and select it.

Sorry if I've misunderstood any part of the problem you've presented, and I'll be glad to help further if I can.

> It also looked to me like in that example at least the modelview was identity.
Yes, the modelview was the identiy in the example.

> In that circumstance, the returned points should be at the near at far planes. In other words, it looks like it's working to me.
Again, I agree.

> You should be able to take those two points and create a selection ray or line segment from them.
Here is where I am having the problem.
I do not know how to find closest points to the picking ray. All my vertices (that I am trying to pick) are in modelview coordinates with z values in the range [9.054, 117.334], but the z coordinate value returned from gluUnProject is in the range [-1, -512]. I don't know how to represent the z coordinate value returned from gluUnProject in terms of my modelview coordinates or vice versa, which I need to do in order to find the distance from one of my points to the picking ray.

Quote:I do not know how to find closest points to the picking ray. All my vertices (that I am trying to pick) are in modelview coordinates with z values in the range [9.054, 117.334], but the z coordinate value returned from gluUnProject is in the range [-1, -512]. I don't know how to represent the z coordinate value returned from gluUnProject in terms of my modelview coordinates or vice versa, which I need to do in order to find the distance from one of my points to the picking ray.

I guess this is where I'm confused. What exactly do you mean by your vertices being 'in modelview coordinates'? To me, that doesn't really mean anything, or more accurately, could mean any number of things.

I can't say for sure, but from the examples you've given I feel like a piece of the puzzle is missing. Your example values indicate the identity modelview matrix, although if this were the case the vertex values you gave wouldn't be visible anyway. So I must be missing something...

If you're still trying to figure this out, perhaps you could post some more of your code. Specifically, it would be useful to see what all transformations, either object or camera related, are being applied to the vertices before rendering. Also, it does make a difference at what point you extract the modelview matrix for submission to gluUnProject(), because that will determine what transforms it contains.

Sorry if any of this seems obvious. I know this technique works; I've implemented it and tested it extensively in my own engine. And although I've never applied the method to individual vertices, I'm confident that the 'distance to ray' method should work. I'm guessing there's just a slight disconnect somewhere that's keeping it from coming together in you case...