Quote:Make your code give you some feedback on whether one of the gluUnProject calls fails.
They return a bool.
When I did that the second one was failing 50% of the time with the above problem.

glUnProject returns an int. I am assuming it returns 0 if it fails?

Quote:I agree with scratt. Make sure the GL and GLU functions work properly (check return values and error states) and check your matrices (the modelview matrix should change when moving the camera, perspectiveand viewport should not - in the normal case).

I was copying the camera's information over the the projection matrix. I altered it to copy over to the model matrix, which I noted was coming in as the identity matrix, so I wasn't over-writing anything.

Quote:do u need to subtract the mouse Y from the viewport height to map from top left origin to OGL's bottom left? I vaguely problems with unproject and it was to do with the z components fed into the 2 unproject calls. draw a line from vec1 to vec2 in OGL every time u click so u can see graphically what u're clicking at. this helped as I found the 2 vector components were quite strange value and i didn't notice when I got it right at first. it was only when I drew the line between the 2 points that I got conformation it was working (perfectly accurate), but if one of ur vecs is always 0,0,2 then I guess they're duff :S

I've been drawing the line the whole time to figure out what is coming out of the function. At this point, it seems to directly correlate to my distance from the center point on the X axis. That is, when I'm in the center, the line is correct. If I go a distance from the center, it goes the right way, but is always off by a bit, as if it is just going a bit too far off to the side. The Y value of the ray is opposite of what it should be, so I multiply it by -1.0f when it comes out of the function.




The problem now is just that it's a little bit off and it's not something I know how to debug.

I've found that when I'm 15% of the screen to the right of the center point (65% across, basically) the point is touching the right side. Appropriately, this correlates directly to when I'm 15% to the left of the center point of the screen (35% across, basically) and it touches the left side.

The same sort of thing is going on with the Y value, which touches the top when the mouse is 20% above the center point (30% of the screen height), and touches the bottom when the mouse is 20% below the center point (70% of the screen height)


EDIT: For now, I've hacked together a small multiplication of around 0.3f for the Right vector and around 0.4 for the Up vector when calculating the picking ray direction. It works for now, but I'll upload the code to see if anyone wants to take a look at it in a minute. Obviously a hack isn't good, but I'd really like to continue with my project so for now it'll do. Thank you all for your help.

Note: I just uprated the guy that helped me in my topic, so the forum won't let me (+rate) you guys yet. Give me a minute. You did help me a lot and I appreciate it.



EDIT AGAIN: Here's the source. The main stuff is in State_MainGame.cpp and it seems to work perfectly now. I'm off to continue my project, but if anyone can find out why I have to use the hack I did, I'd love to know.

http://drewbanyai.net46.net/Files/RayPickingFixed.zip

[Edited by - ShinkaFudan on January 25, 2009 1:45:23 PM]

Quote:Original post by adam17
instead of using the hack, are you making sure that the cursor is at 0,0 when at the top left? i noticed my program had an issue with the borders and the title bar when it came to transforming the mouse coordinates. another problem you could be having is your winZ coordinate could be a little off. does the point on screen line up with your cursor when in the middle, but not line up at the edges of the screen? that would be winZ. if its constantly off by a certain amount then it might be the offsets of the title bar and border.

btw when you move the camera do you get any issues with the zFar and zNear coordinates not wanting to move? (thats why i started this thread)

I checked the four corners and I'm positive that I'm getting (0.0, 0.0) at top left, and (1.0, 1.0) at the bottom right. Since I ended up not using glUnProject, I don't have a WinZ to worry about.

I had a similar problem to yours and fixed it by copying my camera's matrix over to the Projection instead of just taking in what OpenGL gave me... since the camera is what is actually projecting. Try that, if you understand what I mean. If not, send me a PM and I'll try to elaborate (and when you get it working the solution can be posted in the topic, of course).

I don't have that issue. Did you ever solve the problem that you originally posted for? I apologize if I de-railed your topic. It was not my intention.