# gluUnProject stopped working...

This topic is 4326 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I wrote this function a while ago to convert screen coordinates to world coordinates, so that I could then check if the line created by this intersected any of the triangles on the screen, in order to select them:
/*
Converts screen coords to world coords
Inputs:
-XPos, YPos: Screen coords.
-WorldPos1: [Out] A vector containing the actual world coordinates.
-WorldPos2: [Out] A vector containing the world coordinates, but at maximum depth.
Notes:
-Automatically flips the Y coord - 0,0 is the top left like it should be.
-WorldPos2 is provided so that you can easily determine if a given screen coordinate
is overtop of a triangle by forming a line between the two (from the clicked position
to the farthest visible point). WorldPos1 and WorldPos2 can be NULL.
*/
void ScreenToWorldCoords(int XPos, int YPos, Vector* WorldPos1, Vector* WorldPos2)
{
GLint Viewport[4];
GLdouble ModelView[16], Projection[16];
GLfloat XP, YP, ZP;
GLdouble WX, WY, WZ; //World coords this translates to

glGetIntegerv(GL_VIEWPORT, Viewport); //Get the viewport coords - Viewport[] = x, y, width, height
XP = (GLfloat)XPos;
YP = (GLfloat)(Viewport[3] - YPos);
glGetDoublev(GL_MODELVIEW_MATRIX, ModelView); //Get the matrices
glGetDoublev(GL_PROJECTION_MATRIX, Projection);
glReadPixels((GLint)XP, (GLint)YP, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &ZP); //Get the depth of the pixel

if(WorldPos1)
{
gluUnProject((GLdouble)XP, (GLdouble)YP, (GLdouble)ZP, ModelView, Projection, Viewport, &WX, &WY, &WZ); //This gives us the coords of that pixel
(*WorldPos1) = Vector((GLfloat)WX, (GLfloat)WY, (GLfloat)WZ);
}

if(WorldPos2)
{
gluUnProject((GLdouble)XP, (GLdouble)YP, (GLdouble)0.0, ModelView, Projection, Viewport, &WX, &WY, &WZ);
(*WorldPos2) = Vector((GLfloat)WX, (GLfloat)WY, (GLfloat)WZ);
}
}
This was working in an old version, but after adding support for camera rotation, it no longer works even when the camera isn't rotated. The only thing I changed in the drawing code is adding these glRotatef() calls:
glLoadIdentity();

glRotatef(Camera.XRot, 1.0, 0.0, 0.0);
glRotatef(Camera.YRot, 0.0, 1.0, 0.0);
glRotatef(Camera.ZRot, 0.0, 0.0, 1.0);

glTranslatef(Camera.XPos, Camera.YPos, Camera.ZPos);
glTranslatef(this->XPos, this->YPos, this->ZPos);

glBegin(GL_TRIANGLES);
I've checked over everything related to drawing, hit detection and mouse input at least 8 times, compared the source with the old version, etc. I modified both versions to display the two points being returned by ScreenToWorldCoords() as a line between them and the numbers themselves. In the old version the line never even appears (I made sure it's being drawn). In the new version both the line and coordinates clearly indicate the problem: The points returned are wrong. The line that should be projected from the mouse's world position out to the edge of the world is only ever at one of those points - sometimes it comes from way out there to the bottom of the screen, and sometimes it goes from the mouse position, plus an arbitrary depth, straight up. I understand you're supposed to apply any translations, rotations, etc before calling gluUnProject(), and that helped a bit, but still didn't get it working the way it was. The best result I got was by loading the identity matrix into both the modelview and projection matrices, and then repeating all translations and rotations, before calling gluUnProject(). This got correct-looking coordinates - the same X and Y position, following the mouse - but the Z coordinates were always 1.0 and -1.0 instead of Camera.ZPos and the edge of the world like they should be. I checked out the coordinates being returned by both versions under various circumstances: When moving the mouse off the top left corner, where no object exists: New: -92.582, 70.574, -69.654 -1.329, 2.492, -1.000 Old: -1.637, 3.732, -1.232 -1.329, 3.500, -1.000 When pointing at an object around the center of the screen: New: 0.054, 0.740, -4.342 0.013, 1.325, -1.000 Old: -0.030, -0.017, -7.279 -0.004, 2.154, -1.000 They seem to give similar results when pointing at an object, but radically different when not. In fact in the new version I can see the coordinates changing in relation to where the mouse is on the object, and in many cases the line comes down and stops right at its surface, but it doesn't get selected. I just don't understand why... I'd just implemented proper camera rotation, and it was working and then stopped. When I look closer, I can see two large problems... one is that the Z axis, and possibly others, is backward. As I move the mouse along a triangle, up the Z axis, the line is actually moving down the axis. Its coordinates go down, whereas if I move the camera in the same direction, the camera coordinates go up. The second problem is that as soon as the mouse isn't over an object, the X, Y and Z values all seem to be completely random. As I move further up the screen, X and Y jump back and forth between numbers like 62 and 10000 while Z just seems to jump to random 4- or 5-digit numbers. Also, here's the code that tests whether a line intersects a triangle; it's exactly the same in both versions:
/*
Checks if a line intersects a triangle.
Inputs:
-LineStart: Position the line starts at.
-LineEnd: Position the line ends at.
-Tri: Shape to check.
-Point1, Point2, Point3: Points of Tri that form the triangle to test.
Returns: True if the line intersects the triangle, false otherwise.
*/
bool LineIntersectsTriangle(Vector LineStart, Vector LineEnd, SHAPE3D* Tri, unsigned int Point1, unsigned int Point2, unsigned int Point3)
{
Vector V, Intersect, T1, T2, T3, Norm;
GLfloat D1, D2;

V = Tri->Point[Point2].Position - Tri->Point[Point1].Position;
Norm = V.CrossProduct(Tri->Point[Point3].Position - Tri->Point[Point1].Position);
Norm.Normalize();

D1 = (LineStart - Tri->Point[Point1].Position).DotProduct(Norm);
D2 = (LineEnd - Tri->Point[Point1].Position).DotProduct(Norm);
if((D1*D2 < 0.0) //Line crosses triangle
&& (D1 != D2)) //and line and plane aren't parallel - we have a hit
{
Intersect = Vector(LineStart + (LineEnd - LineStart) * (-D1 / (D2 - D1)));

T1 = Norm.CrossProduct(Tri->Point[Point2].Position - Tri->Point[Point1].Position);
T2 = Norm.CrossProduct(Tri->Point[Point3].Position - Tri->Point[Point2].Position);
T3 = Norm.CrossProduct(Tri->Point[Point1].Position - Tri->Point[Point3].Position);

if((T1.DotProduct(Intersect - Tri->Point[Point1].Position) >= 0.0)
&& (T2.DotProduct(Intersect - Tri->Point[Point2].Position) >= 0.0)
&& (T3.DotProduct(Intersect - Tri->Point[Point1].Position) >= 0.0))
return true;
}
return false;
}
The code that handles mouse input and checks if the line intersects any triangles is exactly the same in both versions. The only change between the object drawing code is adding camera rotation, and removing it again doesn't affect anything.  I noticed it works as well as the old version (which really isn't very well), even when the camera is rotated, so long as it stays at position 0,0,0. As soon as the camera moves it all goes out the window. Seems like it's overcompensating for camera movement; if I move the camera up a little, I have to click way below a polygon to hit it. I noticed a signifigant improvement by not loading the identity matrix before calling gluUnProject(), and instead doing this: glTranslatef(-Camera.XPos, -Camera.YPos, -Camera.ZPos); It's still buggy, though. Loading the identity matrix doesn't actually do any good; after doing so, it always just maps the screen coordinates to the coordinates of the viewport itself, not the polygons in it. (If I move the mouse to the left border, the X position is -1.0. The Z positions are always 1.0 and -1.0.) What I suspect is the biggest problem is that it's being called before the object drawing code, relying on the fact that this code doesn't reset the matrices when it's done (since it does so when it starts) and thus just using the data from whatever object was drawn last on the previous frame. Seems like the perfect answer, but again, correcting this has done nothing. >_< [Edited by - HyperHacker on July 22, 2006 5:51:14 AM]

##### Share on other sites
Try normalizing your window coordinates (XP, YP) w.r.t. the center of your screen (ie: XP and YP would be in the range of XP = [-aspect,aspect], Yp = [-1,1], instead of XP = [0,width], YP = [0,height]).

##### Share on other sites
OK, so like this?

XP = ((GLfloat)XPos / (Viewport[2] / 2.0) - 1.0) * ((GLfloat)Viewport[2] / Viewport[3]);YP = (GLfloat)((Viewport[3] / 2.0) - YPos) / (Viewport[3] / 2.0);

That gets me X coordinates between -1.3 and 1.3 and Y coordinates between -1.0 and 1.0 (on a 640x480 window, so aspect is 1.3~). With that, the Z coordinates returned by gluUnProject() appear correct but the X and Y are way off (both around 50-60 - the Z coord is ~52.25 - when they should be close to zero). Also, the coordinates returned are all negated (-52 instead of 52, 60 instead of -60, etc) but it's easy enough to compensate for that.

##### Share on other sites
I typed this in exactly like this and it worked for me:

// transforms up here

int vp[4];
double x,y,z;
float fz;
double M[16],P[16];

glGetDoublev(GL_MODELVIEW_MATRIX,&M[0]);
glGetDoublev(GL_PROJECTION_MATRIX,&P[0]);
glGetIntegerv(GL_VIEWPORT,&vp[0]);
gluUnProject((double)CursorX,(double)(vp[3]-CursorY),(double)fz,&M[0],&P[0],&vp[0],&x,&y,&z);

edit: I noticed on your glReadPixels() call you didn't invert the Y axis as you did with the gluUnProject() call.

I hope that helped :)

##### Share on other sites
Eh? Both calls use YP which is the inverted Y position. Your example is basically what I've been doing.

Here's the latest version, which seems to be getting the Z coord right:
void ScreenToWorldCoords(int XPos, int YPos, Vector* WorldPos1, Vector* WorldPos2){	GLint Viewport[4];	GLdouble ModelView[16], Projection[16];	GLfloat XP, YP, ZP;	GLdouble WX, WY, WZ; //World coords this translates to	glLoadIdentity();	glRotatef(Camera.Rotation.X, 1.0, 0.0, 0.0);	glRotatef(Camera.Rotation.Y, 0.0, 1.0, 0.0);	glRotatef(Camera.Rotation.Z, 0.0, 0.0, 1.0);	glTranslatef(Camera.Position.X, Camera.Position.Y, Camera.Position.Z);	/*glRotatef(Camera.Rotation.X, 1.0, 0.0, 0.0);	glRotatef(Camera.Rotation.Y, 0.0, 1.0, 0.0);	glRotatef(Camera.Rotation.Z, 0.0, 0.0, 1.0);	glTranslatef(-Camera.Position.X, -Camera.Position.Y, -Camera.Position.Z);*/	glGetIntegerv(GL_VIEWPORT, Viewport); //Get the viewport coords - Viewport[] = x, y, width, height	//XP = (GLfloat)XPos / Viewport[2];	XP = ((GLfloat)XPos / (Viewport[2] / 2.0) - 1.0) * ((GLfloat)Viewport[2] / Viewport[3]);	YP = (GLfloat)((Viewport[3] / 2.0) - YPos) / (Viewport[3] / 2.0);	glGetDoublev(GL_MODELVIEW_MATRIX, ModelView); //Get the matrices	glGetDoublev(GL_PROJECTION_MATRIX, Projection);	glReadPixels(XPos, Viewport[3] - YPos, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &ZP); //Get the depth of the pixel	DebugOut(DO_ALL, "Mouse: %d, %d = %2.3f, %2.3f, %2.3f\n", XPos, Viewport[3] - YPos, XP, YP, ZP);	if(WorldPos1)	{		gluUnProject((GLdouble)XP, (GLdouble)YP, (GLdouble)ZP, ModelView, Projection, Viewport, &WX, &WY, &WZ); //This gives us the coords of that pixel		(*WorldPos1) = -Vector((GLfloat)WX, (GLfloat)WY, (GLfloat)WZ);	}	if(WorldPos2)	{		gluUnProject((GLdouble)XP, (GLdouble)YP, (GLdouble)0.0, ModelView, Projection, Viewport, &WX, &WY, &WZ);		(*WorldPos2) = -Vector((GLfloat)WX, (GLfloat)WY, (GLfloat)WZ);	}}

##### Share on other sites
oh... that was actually something I had to correct in mine (duh). (O_<)

Yes, I know it is what you were doing.
I think its weird that it didn't work for you.
The other guy who said normalize your window coords was wrong,
you just directly convert to doubles from ints.

This also worked for me (yes, this is essentially a re-worked verion of yours):

typedef struct tagVECTOR{    double x;    double y;    double z;}vector;void ScreenToWorld(int CX, int CY, vector *p, vector *nzp){    int vp[4];    float fz;    vector V;    double M[16],P[16];    glGetDoublev(GL_MODELVIEW_MATRIX,&M[0]);    glGetDoublev(GL_PROJECTION_MATRIX,&P[0]);    glGetIntegerv(GL_VIEWPORT,&vp[0]);    int iY = vp[3]-CY;    glReadPixels(CX,iY,1,1,GL_DEPTH_COMPONENT,GL_FLOAT,&fz);    if(0 != p)    {        gluUnProject((double)CX,(double)iY,(double)fz,&M[0],&P[0],&vp[0],&V.x,&V.y,&V.z);        *p = V; // the compiler will just copy this temp struct into the one at *p    }    if(0 != nzp)    {        gluUnProject((double)CX,(double)iY,0.0,&M[0],&P[0],&vp[0],&V.x,&V.y,&V.z);        *nzp = V;    }}

Try copy/pasting this. I did and it worked fine.
Again, I hope this helps....

##### Share on other sites
I tried copying that as well as just changing my own to the same thing, in both cases it only worked as well as it did before; works OK (not perfect) until the camera moves. Maybe it has to do with my Vector being an actual class (not just a struct) and using GLfloat instead of double? (I had to add some casting to your code to work with it, but it worked as well as mine. The coordinates I'm working with are actually stored as unsigned shorts, and I have to divide them by 24 to get the real value, so double seems like overkill.)

I keep forgetting to mention, adding rotation didn't break it. For a while it was working just fine no matter where the camera was or how it was rotated. It just stopped when I was fixing other things, I'm not sure when exactly.

Also, it doesn't seem to make any difference whether I translate/rotate to the camera position or not, nor whether I load the identity matrix before doing so. The only thing unusual I noticed is that the position indicator line only appears when the camera is moved, which is when it stops working. I can see the line hitting polygons, and they get highlighted when it does; it's just either starting or ending at the wrong point, so the wrong polygons get hit.

 I made some diagrams of the areas I can select:

This is at position 0,0,0. The polygons in the red boxes are the ones I can select. There's a few others that occasionally become highlighted, some seemingly at random, others at points not on them (for example, I can select one of those black ones at the left by putting the mouse on the white one above it). It almost seems like there's a pattern to the colours that can be selected, but applying textures to them doesn't seem to change anything. Also, I can actually hit a few more in wireframe mode for some reason.

Move the camera up to 0,-1,0 and these polygons are the only ones that ever get hit. Worse, they seem to just get hit randomly, with the cursor position having almost no apparent relation to which one gets hit when. The yellow vertical line at the left is supposed to be showing the ray casted out from the cursor position (Pt1 and Pt2 displayed in the corner are its coordinates) but it's nowhere near the cursor (which has conveniently left a green square on the screen below the line). At the right I circled the area where one polygon got hit once, but I wasn't able to reproduce it.

[Edited by - HyperHacker on July 28, 2006 5:23:52 AM]

##### Share on other sites
I should have asked this beforehand: What compiler are you using?
Does it have a debugger built into it that will allow you to watch
what is happening to the variables?

edit: You shouldn't have to divide by anything to convert a short to an int, or an int to a double. All that happens is the compiler drops the short into the end of a long. That division could very well be the problem.

edit (again): As far as double overkill: There is no need to worry about the number being too big/accurate, only when memory becomes an issue while storing them.

Well, as far as GLfloat goes, its just a typedef of a float (so it's treated the same way), the compiler should make automatic conversions, albeit while complaining at the same time.

As far as the vector class goes, it should be the same as a struct, just with an extra 4 bytes at the begining for type info/vtable pointer added by the compiler (that is if you are either using virtuals in the inheritence chain [if there is one] or have enabled type info), so it isn't the problem.

Here's what I'd do:

For the time being, completely ignore the triangle intersector and anything else not related to extracting and rendering the two points you're interested in.

Here's a layout of what I think your rendering loop should (or might already) look like:

1) Perform ALL global transforms here

2) Render and transform objects here

3) Convert screen coordinates here

4) Draw the two points you're interested in

And if THAT didn't work, I would just very carefully re-write the whole thing using a new set of objects in a separate program.

edit: As in step-by-step, introduce each part of the process until you encounter a problem. Take apart the converter if you have to.

And as a very last resort, I'd design my own camera, avoiding the
glXXX() transform calls completely. However, I cross that bridge ONLY
when I get there.

After you get that working, implementing the intersector should be a breeze.

[Edited by - i-photon on July 28, 2006 9:00:13 PM]

##### Share on other sites
Alright, I think the problem was mostly because I was checking screen coordinates before I drew things. Doing that after instead, the line indicating the two points renders fine, but it's still highlighting the wrong polygons. Here's a nice example:

The blue polygon there is the one being highlighted, while the red and yellow line connects the points that the camera was and the point that was clicked. I can click most of the grass just fine though. In another example that I don't have a picture of, the points of the line are 34.75, 4.373, -79.828 and 16.471, 8.767, -61.0, but the point that gets highlighted is way out at 87.5, 12.5, -200.0. (The camera is around the same place as this image, but the point is way out on those hills at the back.)

The division you mentioned is necessary because the polygon data is stored as unsigned shorts and needs to be scaled down. It's only done when loading the scene to get the proper polygon coordinates that get stored.

BTW, this is when loading the identity matrix and then re-applying the camera translation and rotation. It seems to work when not doing that, though. Also, the line is in the right spot even when the camera is rotated to arbitrary angles on all 3 axes.

##### Share on other sites
Sorry to reply so late... I didn't know you meant the
coordiantes you were talking about were for your triangles, I thought they were
the usual HIWORD() LOWORD() in a DWORD pair that comes from windows.

Do the triangles still seem to get selected at random? I suspect it may be
the intersector in that case. At first sight, yours looks fine. BUT just to be certain, try another one:

[SOURCE]int test_linetri(point &A, point &B, triangle &T){    vector C = A - T.v[0];    vector D = B - A;    vector E1 = T.v[1]-T.v[0];    vector E2 = T.v[0]-T.v[2];    vector N = UnitV(Cross(E1,E2));    float t = Dot(N,D);    if(t > 0.0f) // prevents the line from intersecting triangles                 // "behind" it.    {        t = -Dot(N,C)/t;        D = t*D + T.v[0]; // D is now the intersection        if(Dot(Cross(D-T.v[0],E1),N) >= 0.0f)            if(Dot(Cross(D-T.v[2],E2),N) >= 0.0f)                if(Dot(Cross(D-T.v[2], T.v[2]-T.v[1]),N) >= 0.0f)                    return 1;    }    return 0;}[/SOURCE]

• 38
• 12
• 10
• 10
• 9
• ### Forum Statistics

• Total Topics
631362
• Total Posts
2999577
×