# gluUnProject Translation Problem

This topic is 3294 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I am having an issue with my gluUnProject function. I set it up so i can cast a ray from the near plane to the far plane and not have any problems. I have a point drawn at both ends of the line and they line up perfectly. now if i pan the camera left or right along the x axis there is no problem. if i pan vertically, up or down, along the y axis, i have an issue. the near and far points do not calculate properly. here are some images to prove it. the blue points at the top, bottom, left and right are markers for me. they are placed at x=10, x=-10, y=10, and y=-10. also the camera is at z=10. This is what i get with no movement at all This is if i pan up This is if pan down here is the code for the unproject
//OGLMouseCur, OGLMousePrev, nearPoint, and farPoint are all Vector3 classes
//which contain the normal x, y, and z.
GLint view[4];
GLdouble model[16];
GLdouble proj[16];

glGetDoublev(GL_MODELVIEW_MATRIX, model);
glGetDoublev(GL_PROJECTION_MATRIX, proj);
glGetIntegerv(GL_VIEWPORT, view);

//Copy old position to previous
mousePrev.x = mouseCur.x;
mousePrev.y = mouseCur.y;
OGLMousePrev.x = OGLMouseCur.x;
OGLMousePrev.y = OGLMouseCur.y;
OGLMousePrev.z = OGLMouseCur.z;

//Get new position of cursor
GetCursorPos(&mouseCur);
ScreenToClient(mouseHWND, &mouseCur);
CheckMouseButtons();

//Project cursor coordinates into OGL coordinates
gluUnProject((GLdouble)mouseCur.x-4.0, (GLdouble)(mouseCur.y-30.0), 0.5, model, proj, view,
&OGLMouseCur.x, &OGLMouseCur.y, &OGLMouseCur.z);
gluUnProject((GLdouble)mousePrev.x-4.0, (GLdouble)(mousePrev.y-30.0), 0.5, model, proj, view,
&OGLMousePrev.x, &OGLMousePrev.y, &OGLMousePrev.z);
//Near and far points for line-plane intersection
gluUnProject((GLdouble)mouseCur.x-4.0, (GLdouble)(mouseCur.y-30.0), 1.0, model, proj, view,
&nearPoint.x, &nearPoint.y, &nearPoint.z);
gluUnProject((GLdouble)mouseCur.x-4.0, (GLdouble)(mouseCur.y-30.0), 0.0, model, proj, view,
&farPoint.x, &farPoint.y, &farPoint.z);

OGLMouseCur.y *= -1.0;
OGLMousePrev.y *= -1.0;
nearPoint.y *= -1.0f;
farPoint.y *= -1.0f;


does anybody have an idea of why this is happening? thank you! [Edited by - adam17 on January 17, 2009 4:43:05 PM]

##### Share on other sites
all the extra additions to the numbers youre doing seems a bit hacky

if u want a ray through the cursor pos u only need to unproject the following 2 points

gluUnProject((GLdouble)mouseCur.x, (GLdouble)(mouseCur.y), 0.0, model, proj, view,
&n.x, &n.y, &n.z);

gluUnProject((GLdouble)mouseCur.x, (GLdouble)(mouseCur.y), 1.0, model, proj, view,
&f.x, &f.y, &f.z);

what r u wanting to do with this ray?

though if u draw a line between these points u wont be able to see it (picture looking along a line)

##### Share on other sites
Maybe those numbers are some sort of hot spot (i.e. the "click point" of the cursor), but that still would be better done in a less hardcoded way :)

Also, why do you first save OGLMouseCur to OGLMousePrev and then overwrite it by unprojecting the previous mouse position into OGLMousePrev? That operation should be unnecessary and would likely cause OGLMousePrev to change if the modelview matrix changed since the last frame.

Next, remember that z = 1.0 represents the far plane, not the near plane as indicated in your code. You should swap the 1.0 and 0.0 in those two lines (3rd and 4th gluUnproject().

##### Share on other sites
Quote:
 Original post by zedzall the extra additions to the numbers youre doing seems a bit hackyif u want a ray through the cursor pos u only need to unproject the following 2 pointsgluUnProject((GLdouble)mouseCur.x, (GLdouble)(mouseCur.y), 0.0, model, proj, view, &n.x, &n.y, &n.z);gluUnProject((GLdouble)mouseCur.x, (GLdouble)(mouseCur.y), 1.0, model, proj, view, &f.x, &f.y, &f.z);what r u wanting to do with this ray?though if u draw a line between these points u wont be able to see it (picture looking along a line)

yeah my code is kinda hacky right now. there are a lot of extras and unneeded variables, but they will be filtered out once i get this solved.

im trying to build a 2d/3d world editor. i realize i will not be able to see the line, except for directly onto it. im only drawing the line to aid in figuring out where my 2 points are, even if one is out of sight. im using the line for ray tracing the scene and selecting objects, a WYSIWYG editor.

Quote:
 Original post by Lord_EvilMaybe those numbers are some sort of hot spot (i.e. the "click point" of the cursor), but that still would be better done in a less hardcoded way :)Also, why do you first save OGLMouseCur to OGLMousePrev and then overwrite it by unprojecting the previous mouse position into OGLMousePrev? That operation should be unnecessary and would likely cause OGLMousePrev to change if the modelview matrix changed since the last frame.Next, remember that z = 1.0 represents the far plane, not the near plane as indicated in your code. You should swap the 1.0 and 0.0 in those two lines (3rd and 4th gluUnproject().

where are you talking about "click point" numbers? the -30 and the -4? i know there is a function that can get the coordinates of the window, but i cant remember it, nor find it.

i fixed the OGLMousePrev overwrite. i removed it.

im a little confused on how z=0 is the near plane and z=1 is the far plane. on screen, it shows the opposite. the color coded points are the give away. although, it may be a reason for my problem, but i cant figure it out.

##### Share on other sites
The z-component represents the depth. 1.0 means full depth, i.e. farthest position, 0.0 means no depth/nearest position.

The pictures don't show up in my browser (firefox 3), so I can only guess (maybe you could also provide the links). Whether a fragment (pixel) overwrites another one depends on the depth and the depth function or - if depth testing is disabled - the rendering order. If you have depth testing enabled but changed your depth comparison to something like GL_GREATER, farther away fragments overwrite nearer ones.

##### Share on other sites
Lord_Evil you are completely and totally right about the depth values. i was confused at first and googled depth maps to figure out if white was near or far. well, i found a bunch of images with white on the near plane. come to find out, they are used in 3d images, not depth testing. i thank you for pointing the 1.0/0.0 issue out to me.

i have uploaded the images to flickr and now they should show up. to add on to my images, the mouse cursor is hanging on to the blue dot, which is the far point. what is going on?! my only guess is that the unproject function is somehow not taking into consideration the translation of the camera. the nearpoint jumps off of the screen quickly, due to the nearplane being stuck at the original position. therefore when i pan, it moves a lot quicker.

[Edited by - adam17 on January 20, 2009 6:58:13 PM]

##### Share on other sites
i should also point out that i am using a perspective projection rather than an ortho projection. im also using gluLookAt. i could not find out whether or not gluLookAt modifies the ModelView matrix. any ideas?

##### Share on other sites
With the information gathered from this topic, I tried to create a function that would give me a 3-float vector representing the direction of a ray from the camera to a point I was clicking on some geometry in-game. I was trying to get ray-picking done through this method.

I came up with this:

vec3f CalculatePickingRay2(unsigned int ScreenX, unsigned int ScreenY){	vec3d Point1;	vec3d Point2;	GLint view[4];	GLdouble model[16];	GLdouble proj[16];	glGetDoublev(GL_MODELVIEW_MATRIX, model);	glGetDoublev(GL_PROJECTION_MATRIX, proj);	glGetIntegerv(GL_VIEWPORT, view);	gluUnProject((GLdouble)ScreenX, (GLdouble)ScreenY, 0.0, model, proj, view, &(Point1.x), &(Point1.y), &(Point1.z));	gluUnProject((GLdouble)ScreenX, (GLdouble)ScreenY, 1.0, model, proj, view, &(Point2.x), &(Point2.y), &(Point2.z));	return vec3f((float)(Point1.x - Point2.x), (float)(Point1.y - Point2.y), (float)(Point1.z - Point2.z));}

I must be doing something wrong, though.... because the vector I get back is always (no matter if I rotate or translate the camera different ways) returns a vec3f that is (0.0f, 0.0f, 2.0f)

Any help getting back on the right track would be greatly appreciated.

##### Share on other sites
Quote:
 Original post by ShinkaFudanWith the information gathered from this topic, I tried to create a function that would give me a 3-float vector representing the direction of a ray from the camera to a point I was clicking on some geometry in-game. I was trying to get ray-picking done through this method.I came up with this:vec3f CalculatePickingRay2(unsigned int ScreenX, unsigned int ScreenY){ vec3d Point1; vec3d Point2; GLint view[4]; GLdouble model[16]; GLdouble proj[16]; glGetDoublev(GL_MODELVIEW_MATRIX, model); glGetDoublev(GL_PROJECTION_MATRIX, proj); glGetIntegerv(GL_VIEWPORT, view); gluUnProject((GLdouble)ScreenX, (GLdouble)ScreenY, 0.0, model, proj, view, &(Point1.x), &(Point1.y), &(Point1.z)); gluUnProject((GLdouble)ScreenX, (GLdouble)ScreenY, 1.0, model, proj, view, &(Point2.x), &(Point2.y), &(Point2.z)); return vec3f((float)(Point1.x - Point2.x), (float)(Point1.y - Point2.y), (float)(Point1.z - Point2.z));}I must be doing something wrong, though.... because the vector I get back is always (no matter if I rotate or translate the camera different ways) returns a vec3f that is (0.0f, 0.0f, 2.0f)Any help getting back on the right track would be greatly appreciated.

At a glance your code looks fine to me. Very similar to my method. But I also had problems getting it to work initially. Do just check if you should be taking near from far, or far from near. I don't have my code in-front of me and I cannot remember to be honest. But you may have that swapped.

One problem I didn't have, but read about in my travels solving my problems was that you need to make sure that your GL matrices are valid when you retrieve them. I do that in a known location in my code and then reference them from there for everything.

Another is to make sure the orientation of your screen coordinates run from the correct origin.
Some systems y is swapped.

The problem I had was that the zfar at 1.0 gave me a lot of failures. I moved it in a bit, and that solved it. I still don't know why. Which niggles me. But it's solved for now. Eventually I will re-write gluUnProject as part of my own libraries.

Make your code give you some feedback on whether one of the gluUnProject calls fails.
They return a bool.
When I did that the second one was failing 50% of the time with the above problem.

##### Share on other sites
I agree with scratt. Make sure the GL and GLU functions work properly (check return values and error states) and check your matrices (the modelview matrix should change when moving the camera, perspectiveand viewport should not - in the normal case).

##### Share on other sites
do u need to subtract the mouse Y from the viewport height to map from top left origin to OGL's bottom left? I vaguely problems with unproject and it was to do with the z components fed into the 2 unproject calls. draw a line from vec1 to vec2 in OGL every time u click so u can see graphically what u're clicking at. this helped as I found the 2 vector components were quite strange value and i didn't notice when I got it right at first. it was only when I drew the line between the 2 points that I got conformation it was working (perfectly accurate), but if one of ur vecs is always 0,0,2 then I guess they're duff :S

##### Share on other sites
Quote:
 Make your code give you some feedback on whether one of the gluUnProject calls fails.They return a bool.When I did that the second one was failing 50% of the time with the above problem.

glUnProject returns an int. I am assuming it returns 0 if it fails?

Quote:
 I agree with scratt. Make sure the GL and GLU functions work properly (check return values and error states) and check your matrices (the modelview matrix should change when moving the camera, perspectiveand viewport should not - in the normal case).

I was copying the camera's information over the the projection matrix. I altered it to copy over to the model matrix, which I noted was coming in as the identity matrix, so I wasn't over-writing anything.

Quote:
 do u need to subtract the mouse Y from the viewport height to map from top left origin to OGL's bottom left? I vaguely problems with unproject and it was to do with the z components fed into the 2 unproject calls. draw a line from vec1 to vec2 in OGL every time u click so u can see graphically what u're clicking at. this helped as I found the 2 vector components were quite strange value and i didn't notice when I got it right at first. it was only when I drew the line between the 2 points that I got conformation it was working (perfectly accurate), but if one of ur vecs is always 0,0,2 then I guess they're duff :S

I've been drawing the line the whole time to figure out what is coming out of the function. At this point, it seems to directly correlate to my distance from the center point on the X axis. That is, when I'm in the center, the line is correct. If I go a distance from the center, it goes the right way, but is always off by a bit, as if it is just going a bit too far off to the side. The Y value of the ray is opposite of what it should be, so I multiply it by -1.0f when it comes out of the function.

The problem now is just that it's a little bit off and it's not something I know how to debug.

I've found that when I'm 15% of the screen to the right of the center point (65% across, basically) the point is touching the right side. Appropriately, this correlates directly to when I'm 15% to the left of the center point of the screen (35% across, basically) and it touches the left side.

The same sort of thing is going on with the Y value, which touches the top when the mouse is 20% above the center point (30% of the screen height), and touches the bottom when the mouse is 20% below the center point (70% of the screen height)

EDIT: For now, I've hacked together a small multiplication of around 0.3f for the Right vector and around 0.4 for the Up vector when calculating the picking ray direction. It works for now, but I'll upload the code to see if anyone wants to take a look at it in a minute. Obviously a hack isn't good, but I'd really like to continue with my project so for now it'll do. Thank you all for your help.

Note: I just uprated the guy that helped me in my topic, so the forum won't let me (+rate) you guys yet. Give me a minute. You did help me a lot and I appreciate it.

EDIT AGAIN: Here's the source. The main stuff is in State_MainGame.cpp and it seems to work perfectly now. I'm off to continue my project, but if anyone can find out why I have to use the hack I did, I'd love to know.

http://drewbanyai.net46.net/Files/RayPickingFixed.zip

[Edited by - ShinkaFudan on January 25, 2009 1:45:23 PM]

##### Share on other sites
instead of using the hack, are you making sure that the cursor is at 0,0 when at the top left? i noticed my program had an issue with the borders and the title bar when it came to transforming the mouse coordinates. another problem you could be having is your winZ coordinate could be a little off. does the point on screen line up with your cursor when in the middle, but not line up at the edges of the screen? that would be winZ. if its constantly off by a certain amount then it might be the offsets of the title bar and border.

btw when you move the camera do you get any issues with the zFar and zNear coordinates not wanting to move? (thats why i started this thread)

##### Share on other sites
Quote:
 Original post by adam17instead of using the hack, are you making sure that the cursor is at 0,0 when at the top left? i noticed my program had an issue with the borders and the title bar when it came to transforming the mouse coordinates. another problem you could be having is your winZ coordinate could be a little off. does the point on screen line up with your cursor when in the middle, but not line up at the edges of the screen? that would be winZ. if its constantly off by a certain amount then it might be the offsets of the title bar and border. btw when you move the camera do you get any issues with the zFar and zNear coordinates not wanting to move? (thats why i started this thread)

I checked the four corners and I'm positive that I'm getting (0.0, 0.0) at top left, and (1.0, 1.0) at the bottom right. Since I ended up not using glUnProject, I don't have a WinZ to worry about.

I had a similar problem to yours and fixed it by copying my camera's matrix over to the Projection instead of just taking in what OpenGL gave me... since the camera is what is actually projecting. Try that, if you understand what I mean. If not, send me a PM and I'll try to elaborate (and when you get it working the solution can be posted in the topic, of course).

I don't have that issue. Did you ever solve the problem that you originally posted for? I apologize if I de-railed your topic. It was not my intention.

##### Share on other sites
well i added a bit of code to output the modelview matrix along with the near and farpoints into a log file. here is what i came up with:

//Quad INear: 0.100278 0.00972222 9.9Far:  100.379 9.73207 -90.1013ModelView Matrix    1     0     0     0     0     1     0     0     0     0     1     0     0     0   -10     1 //Quad IINear: -0.100278 0.01 9.9Far:  -100.379 10.0101 -90.1013ModelView Matrix    1     0     0     0     0     1     0     0     0     0     1     0     0     0   -10     1 //Quad IIINear: -0.100278 -0.01 9.9Far:  -100.379 -10.0101 -90.1013ModelView Matrix    1     0     0     0     0     1     0     0     0     0     1     0     0     0   -10     1 Quad IVNear: 0.1 -0.00972222 9.9Far:  100.101 -9.73207 -90.1013ModelView Matrix    1     0     0     0     0     1     0     0     0     0     1     0     0     0   -10     1 //panned up, Near.x is correct, Near.y is wrong. Far is correct.Near: 0.01 -7.96389 9.9Far:  10.0101 28.1477 -90.1013ModelView Matrix    1     0     0     0     0     1     0     0     0     0     1     0     0    -8   -10     1 //panned down, Near.x is correct, Near.y is wrong. Far is correctNear: 0.01 7.96417 9.9Far:  10.0101 -27.8696 -90.1013ModelView Matrix    1     0     0     0     0     1     0     0     0     0     1     0     0     8   -10     1

the modelview matrix is correct, but the nears is off. all of the coordinates are correct except for the Near point's y value after panning.

any ideas?

[Edited by - adam17 on February 17, 2009 8:50:55 PM]