Maybe those numbers are some sort of hot spot (i.e. the "click point" of the cursor), but that still would be better done in a less hardcoded way :)

Also, why do you first save OGLMouseCur to OGLMousePrev and then overwrite it by unprojecting the previous mouse position into OGLMousePrev? That operation should be unnecessary and would likely cause OGLMousePrev to change if the modelview matrix changed since the last frame.

Next, remember that z = 1.0 represents the far plane, not the near plane as indicated in your code. You should swap the 1.0 and 0.0 in those two lines (3rd and 4th gluUnproject().

The z-component represents the depth. 1.0 means full depth, i.e. farthest position, 0.0 means no depth/nearest position.

The pictures don't show up in my browser (firefox 3), so I can only guess (maybe you could also provide the links). Whether a fragment (pixel) overwrites another one depends on the depth and the depth function or - if depth testing is disabled - the rendering order. If you have depth testing enabled but changed your depth comparison to something like GL_GREATER, farther away fragments overwrite nearer ones.

With the information gathered from this topic, I tried to create a function that would give me a 3-float vector representing the direction of a ray from the camera to a point I was clicking on some geometry in-game. I was trying to get ray-picking done through this method.

I came up with this:

vec3f CalculatePickingRay2(unsigned int ScreenX, unsigned int ScreenY){	vec3d Point1;	vec3d Point2;	GLint view[4];	GLdouble model[16];	GLdouble proj[16];	glGetDoublev(GL_MODELVIEW_MATRIX, model);	glGetDoublev(GL_PROJECTION_MATRIX, proj);	glGetIntegerv(GL_VIEWPORT, view);	gluUnProject((GLdouble)ScreenX, (GLdouble)ScreenY, 0.0, model, proj, view, &(Point1.x), &(Point1.y), &(Point1.z));	gluUnProject((GLdouble)ScreenX, (GLdouble)ScreenY, 1.0, model, proj, view, &(Point2.x), &(Point2.y), &(Point2.z));	return vec3f((float)(Point1.x - Point2.x), (float)(Point1.y - Point2.y), (float)(Point1.z - Point2.z));}

I must be doing something wrong, though.... because the vector I get back is always (no matter if I rotate or translate the camera different ways) returns a vec3f that is (0.0f, 0.0f, 2.0f)

Any help getting back on the right track would be greatly appreciated.

Quote:Original post by ShinkaFudan
With the information gathered from this topic, I tried to create a function that would give me a 3-float vector representing the direction of a ray from the camera to a point I was clicking on some geometry in-game. I was trying to get ray-picking done through this method.

I came up with this:

vec3f CalculatePickingRay2(unsigned int ScreenX, unsigned int ScreenY){	vec3d Point1;	vec3d Point2;	GLint view[4];	GLdouble model[16];	GLdouble proj[16];	glGetDoublev(GL_MODELVIEW_MATRIX, model);	glGetDoublev(GL_PROJECTION_MATRIX, proj);	glGetIntegerv(GL_VIEWPORT, view);	gluUnProject((GLdouble)ScreenX, (GLdouble)ScreenY, 0.0, model, proj, view, &(Point1.x), &(Point1.y), &(Point1.z));	gluUnProject((GLdouble)ScreenX, (GLdouble)ScreenY, 1.0, model, proj, view, &(Point2.x), &(Point2.y), &(Point2.z));	return vec3f((float)(Point1.x - Point2.x), (float)(Point1.y - Point2.y), (float)(Point1.z - Point2.z));}

I must be doing something wrong, though.... because the vector I get back is always (no matter if I rotate or translate the camera different ways) returns a vec3f that is (0.0f, 0.0f, 2.0f)

Any help getting back on the right track would be greatly appreciated.

At a glance your code looks fine to me. Very similar to my method. But I also had problems getting it to work initially. Do just check if you should be taking near from far, or far from near. I don't have my code in-front of me and I cannot remember to be honest. But you may have that swapped.

One problem I didn't have, but read about in my travels solving my problems was that you need to make sure that your GL matrices are valid when you retrieve them. I do that in a known location in my code and then reference them from there for everything.

Another is to make sure the orientation of your screen coordinates run from the correct origin.
Some systems y is swapped.

The problem I had was that the zfar at 1.0 gave me a lot of failures. I moved it in a bit, and that solved it. I still don't know why. Which niggles me. But it's solved for now. Eventually I will re-write gluUnProject as part of my own libraries.

Make your code give you some feedback on whether one of the gluUnProject calls fails.
They return a bool.
When I did that the second one was failing 50% of the time with the above problem.

I agree with scratt. Make sure the GL and GLU functions work properly (check return values and error states) and check your matrices (the modelview matrix should change when moving the camera, perspectiveand viewport should not - in the normal case).