I wrote this function a while ago to convert screen coordinates to world coordinates, so that I could then check if the line created by this intersected any of the triangles on the screen, in order to select them:
/*
Converts screen coords to world coords
Inputs:
-XPos, YPos: Screen coords.
-WorldPos1: [Out] A vector containing the actual world coordinates.
-WorldPos2: [Out] A vector containing the world coordinates, but at maximum depth.
Notes:
-Automatically flips the Y coord - 0,0 is the top left like it should be.
-WorldPos2 is provided so that you can easily determine if a given screen coordinate
is overtop of a triangle by forming a line between the two (from the clicked position
to the farthest visible point). WorldPos1 and WorldPos2 can be NULL.
*/
void ScreenToWorldCoords(int XPos, int YPos, Vector* WorldPos1, Vector* WorldPos2)
{
GLint Viewport[4];
GLdouble ModelView[16], Projection[16];
GLfloat XP, YP, ZP;
GLdouble WX, WY, WZ; //World coords this translates to
glGetIntegerv(GL_VIEWPORT, Viewport); //Get the viewport coords - Viewport[] = x, y, width, height
XP = (GLfloat)XPos;
YP = (GLfloat)(Viewport[3] - YPos);
glGetDoublev(GL_MODELVIEW_MATRIX, ModelView); //Get the matrices
glGetDoublev(GL_PROJECTION_MATRIX, Projection);
glReadPixels((GLint)XP, (GLint)YP, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &ZP); //Get the depth of the pixel
if(WorldPos1)
{
gluUnProject((GLdouble)XP, (GLdouble)YP, (GLdouble)ZP, ModelView, Projection, Viewport, &WX, &WY, &WZ); //This gives us the coords of that pixel
(*WorldPos1) = Vector((GLfloat)WX, (GLfloat)WY, (GLfloat)WZ);
}
if(WorldPos2)
{
gluUnProject((GLdouble)XP, (GLdouble)YP, (GLdouble)0.0, ModelView, Projection, Viewport, &WX, &WY, &WZ);
(*WorldPos2) = Vector((GLfloat)WX, (GLfloat)WY, (GLfloat)WZ);
}
}
This was working in an old version, but after adding support for camera rotation, it no longer works even when the camera isn't rotated. The only thing I changed in the drawing code is adding these glRotatef() calls:
glLoadIdentity();
glRotatef(Camera.XRot, 1.0, 0.0, 0.0);
glRotatef(Camera.YRot, 0.0, 1.0, 0.0);
glRotatef(Camera.ZRot, 0.0, 0.0, 1.0);
glTranslatef(Camera.XPos, Camera.YPos, Camera.ZPos);
glTranslatef(this->XPos, this->YPos, this->ZPos);
glBegin(GL_TRIANGLES);
I've checked over everything related to drawing, hit detection and mouse input at least 8 times, compared the source with the old version, etc. I modified both versions to display the two points being returned by ScreenToWorldCoords() as a line between them and the numbers themselves. In the old version the line never even appears (I made sure it's being drawn). In the new version both the line and coordinates clearly indicate the problem: The points returned are wrong. The line that should be projected from the mouse's world position out to the edge of the world is only ever at one of those points - sometimes it comes from way out there to the bottom of the screen, and sometimes it goes from the mouse position, plus an arbitrary depth, straight up.
I understand you're supposed to apply any translations, rotations, etc before calling gluUnProject(), and that helped a bit, but still didn't get it working the way it was. The best result I got was by loading the identity matrix into both the modelview and projection matrices, and then repeating all translations and rotations, before calling gluUnProject(). This got correct-looking coordinates - the same X and Y position, following the mouse - but the Z coordinates were always 1.0 and -1.0 instead of Camera.ZPos and the edge of the world like they should be.
I checked out the coordinates being returned by both versions under various circumstances:
When moving the mouse off the top left corner, where no object exists:
New:
-92.582, 70.574, -69.654
-1.329, 2.492, -1.000
Old:
-1.637, 3.732, -1.232
-1.329, 3.500, -1.000
When pointing at an object around the center of the screen:
New:
0.054, 0.740, -4.342
0.013, 1.325, -1.000
Old:
-0.030, -0.017, -7.279
-0.004, 2.154, -1.000
They seem to give similar results when pointing at an object, but radically different when not. In fact in the new version I can see the coordinates changing in relation to where the mouse is on the object, and in many cases the line comes down and stops right at its surface, but it doesn't get selected. I just don't understand why... I'd just implemented proper camera rotation, and it was working and then stopped.
When I look closer, I can see two large problems... one is that the Z axis, and possibly others, is backward. As I move the mouse along a triangle, up the Z axis, the line is actually moving
down the axis. Its coordinates go down, whereas if I move the camera in the same direction, the camera coordinates go up. The second problem is that as soon as the mouse isn't over an object, the X, Y and Z values all seem to be completely random. As I move further up the screen, X and Y jump back and forth between numbers like 62 and 10000 while Z just seems to jump to random 4- or 5-digit numbers.
Also, here's the code that tests whether a line intersects a triangle; it's exactly the same in both versions:
/*
Checks if a line intersects a triangle.
Inputs:
-LineStart: Position the line starts at.
-LineEnd: Position the line ends at.
-Tri: Shape to check.
-Point1, Point2, Point3: Points of Tri that form the triangle to test.
Returns: True if the line intersects the triangle, false otherwise.
*/
bool LineIntersectsTriangle(Vector LineStart, Vector LineEnd, SHAPE3D* Tri, unsigned int Point1, unsigned int Point2, unsigned int Point3)
{
Vector V, Intersect, T1, T2, T3, Norm;
GLfloat D1, D2;
V = Tri->Point[Point2].Position - Tri->Point[Point1].Position;
Norm = V.CrossProduct(Tri->Point[Point3].Position - Tri->Point[Point1].Position);
Norm.Normalize();
D1 = (LineStart - Tri->Point[Point1].Position).DotProduct(Norm);
D2 = (LineEnd - Tri->Point[Point1].Position).DotProduct(Norm);
if((D1*D2 < 0.0) //Line crosses triangle
&& (D1 != D2)) //and line and plane aren't parallel - we have a hit
{
Intersect = Vector(LineStart + (LineEnd - LineStart) * (-D1 / (D2 - D1)));
T1 = Norm.CrossProduct(Tri->Point[Point2].Position - Tri->Point[Point1].Position);
T2 = Norm.CrossProduct(Tri->Point[Point3].Position - Tri->Point[Point2].Position);
T3 = Norm.CrossProduct(Tri->Point[Point1].Position - Tri->Point[Point3].Position);
if((T1.DotProduct(Intersect - Tri->Point[Point1].Position) >= 0.0)
&& (T2.DotProduct(Intersect - Tri->Point[Point2].Position) >= 0.0)
&& (T3.DotProduct(Intersect - Tri->Point[Point1].Position) >= 0.0))
return true;
}
return false;
}
The code that handles mouse input and checks if the line intersects any triangles is exactly the same in both versions. The only change between the object drawing code is adding camera rotation, and removing it again doesn't affect anything.
[edit] I noticed it works as well as the old version (which really isn't very well), even when the camera is rotated, so long as it stays at position 0,0,0. As soon as the camera moves it all goes out the window. Seems like it's overcompensating for camera movement; if I move the camera up a little, I have to click way below a polygon to hit it. I noticed a signifigant improvement by not loading the identity matrix before calling gluUnProject(), and instead doing this:
glTranslatef(-Camera.XPos, -Camera.YPos, -Camera.ZPos);
It's still buggy, though. Loading the identity matrix doesn't actually do any good; after doing so, it always just maps the screen coordinates to the coordinates of the viewport itself, not the polygons in it. (If I move the mouse to the left border, the X position is -1.0. The Z positions are always 1.0 and -1.0.)
What I suspect is the biggest problem is that it's being called before the object drawing code, relying on the fact that this code doesn't reset the matrices when it's done (since it does so when it starts) and thus just using the data from whatever object was drawn last on the previous frame. Seems like the perfect answer, but again, correcting this has done nothing. >_<
[Edited by - HyperHacker on July 22, 2006 5:51:14 AM]
