Im trying to make an rts, which is using 3d terrain which can be viewed from any angle. My problem is figureing out what the mouse was clicked on. Right now Im casting a ray into the screen from the mouse, but the problem is that it is not straight back it is being affected by perspective. My question is how I can get it to be cast strait back. I can send the code to those who ask. Thanks allot

the real question is... what do you want to do with your mouse ?
to you want to know on where it hits the terrain, or where it hits any object in the 3D view ?
to you want to retrieve coordinates of the terrain, assuming that the terrain is a DEM (eg, defined in x/z with altitudes in y) ?

You can use the selection buffer, but the way you have to use it depends on what you look for.

I was trying to avoid the selection buffer, its slow (on my system anyway). I want to be able find where on the terrain it is clicked. The way it works now, is I convert the x&y coords from get cursor pos and the from that point i cast the ray back.

I want to know what is under that point

Thnx

I have one solution, but may not be the best :
under the mouse position, get the depth value (glReadPixels on the Depth Buffer) to get the 3rd coordinate.

You have X and Y coordinates (thanks to mouse coordinates) and now Z.
You *just* have to use some mathematical formula to use the projection and modelview matrix to retrive your coordinate in the 3D world from the "screen 3D coordinate".

Hope this helps.

Thats a good idea... Now to figure out how todo the second part

actually reading from the depth buffer to get the z coordinate will not work to well. you have to realize that the x,y you get from the GetCursorPos() function should be considered trasnformed points. this means that the x/y will not match the z in the depth buffer due to perspetive rendering. (the z buffer method may work in certain cases, but i have a strong feeling that it dont work in all cases like the ray cast method you are using).

condsider the problem like this: you have an x/y pair of transformed vertices. you require a vector going from the transformed pair along the viewing vector. this should give you the desired result. since the camera position is always at the center (0,0,0) after all the transformations are done, you can use that fact to help get the pre transformed line (since it will be a line along the z plane. i will warn you right now that i have not tested this theory, but it is pretty sound. here is the algo, if you care to try it:
1. get the transformed x/y
2. back transform your x/y mouse coordinates (there should be some UnProject function in gl since there is one in d3d, if not just apply you transformations in reverse order AND inversed).
3. now you have an x/y pair (and a psudeo z). now cast yoru ray along that line (ie the z value should only be changing). unfortunatly i am not sure exactly how similar that is to your system (i used this technique for draw particles where the mouse was located). you may need to cast the ray from the back transformed x,y,z to the x,y,z of the camera position in world space. (make sure on the back transformation you keep your back transformed mouse coordinates in world space). the vector that is created between those should be what you are looking for. then its just a matter of finding the first object it collides with closest to the camera and in front of it.


EDIT: heh, now that i think about it, the zbuffer are transformed points as well. the zbuffer method should work just as well if not better.

Edited by - a person on February 4, 2002 5:27:04 PM

Ok... well this way seems to be too complicated... so idecided id just f--- it and go with selection. I just cant seem to get it to work, because I draw a number of triangle strips.
the code is as follows:

  	int count=0;	  for(int y = 2;y<YW-3;y++) //Smoothing wont fix the edges... so chop em off!  { //This gets me an amazing 150 fps with 128*128 terrain!	  glBegin(GL_TRIANGLE_STRIP);	  for(int x = 2;x<XW-2;x++)	  { 		if(!selection && PointInFrustum(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z,11))		{		  glTexCoord2d(1,1);glNormal3f(normals[x][y+1].x,normals[x][y+1].y,normals[x][y+1].z); glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);		  glTexCoord2d(1,0);glNormal3f(normals[x][y].x,normals[x][y].y,normals[x][y].z);glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);  		  glTexCoord2d(0,1);glNormal3f(normals[x][y+1].x,normals[x][y+1].y,normals[x][y+1].z);glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);		  glTexCoord2d(0,0);glNormal3f(normals[x][y].x,normals[x][y].y,normals[x][y].z);glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);		  polies++;		}		if(selection)		{		  glPushName(count);		  glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);		  glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);		}		count++;	  }  	  glEnd();  }  

I cant seem to get it to work, the only hit i can ever get is the one I push onto the stack before i begin the selection.
Any suggestions?

Ok... well this way seems to be too complicated... so idecided id just f--- it and go with selection. I just cant seem to get it to work, because I draw a number of triangle strips.
the code is as follows:

  	int count=0;	  for(int y = 2;y<YW-3;y++) //Smoothing wont fix the edges... so chop em off!  { //This gets me an amazing 150 fps with 128*128 terrain!	  glBegin(GL_TRIANGLE_STRIP);	  for(int x = 2;x<XW-2;x++)	  { 		if(!selection && PointInFrustum(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z,11))		{		  glTexCoord2d(1,1);glNormal3f(normals[x][y+1].x,normals[x][y+1].y,normals[x][y+1].z); glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);		  glTexCoord2d(1,0);glNormal3f(normals[x][y].x,normals[x][y].y,normals[x][y].z);glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);  		  glTexCoord2d(0,1);glNormal3f(normals[x][y+1].x,normals[x][y+1].y,normals[x][y+1].z);glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);		  glTexCoord2d(0,0);glNormal3f(normals[x][y].x,normals[x][y].y,normals[x][y].z);glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);		  polies++;		}		if(selection)		{		  glPushName(count);		  glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);		  glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);		}		count++;	  }  	  glEnd();  }  

I cant seem to get it to work, the only hit i can ever get is the one I push onto the stack before i begin the selection.
Any suggestions?

I don't know if you already knew, but you gotta use "picking".

#define BUFSIZE 512
void mouse_click(int x, int y)
{
GLuint selectBuf[BUFSIZE];
GLint hits;
GLint viewport[4];

// Get window size.
glGetIntegerv (GL_VIEWPORT, viewport);

// Set select buffer
glSelectBuffer (BUFSIZE, selectBuf);

// Switch to rendering mode to selection. Nothing will be drawn on screen
(void)glRenderMode(GL_SELECT);

// Initialize name stack
glInitNames();
glPushName(0);

// Set the projection matrix to focus around mouse cooridnates
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();
/* create 5x5 pixel picking region near cursor location */
gluPickMatrix((GLdouble)x, (GLdouble)(viewport[3] - y), 5.0, 5.0, viewport);
gluPrespective(...); // Copy the command that you use for your perspective or ortho.

// Render the scene
glMatrixMode(GL_MODELVIEW);
draw_my_scene(GL_SELECT);

// Restore projection matrix
glMatrixMode(GL_PROJECTION);
glPopMatrix();

// Switch to normal rendering mode, and get the number of hits in the selection buffer
glFlush();
hits = glRenderMode(GL_RENDER);

// Do what you want with the selection buffer
process_hits(hits, selectBuf);
}



and you scene may be draw with something like that :

void draw_my_scene(GLint rendering_mode = GL_RENDER)
{
bool selection = (rendering_mode==GL_SELECT);

if(selection)
for(int y = 2;y < YW-3;y++) //Smoothing wont fix the edges... so chop em off!
{ //This gets me an amazing 150 fps with 128*128 terrain!
glLoadName(y);
for(int x = 3;x < XW-2;x++)
{
glPushName(x);
glBegin(GL_TRIANGLE_STRIP);
glVertex3f(verts[x-1][y+1].x,verts[x-1][y+1].y,verts[x-1][y+1].z);
glVertex3f(verts[x-1][y].x,verts[x-1][y].y,verts[x-1][y].z);
glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);
glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);
glEnd();
glPopName();
}
}
else // eg (!selection)
for(int y = 2;y < YW-3;y++) //Smoothing wont fix the edges... so chop em off!
{ //This gets me an amazing 150 fps with 128*128 terrain!
glBegin(GL_TRIANGLE_STRIP);
for(int x = 2;x < XW-2;x++)
{
if(PointInFrustum(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z,11))
{
glTexCoord2d(1,1);glNormal3f(normals[x][y+1].x,normals[x][y+1].y,normals[x][y+1].z);glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);
glTexCoord2d(1,0);glNormal3f(normals[x][y].x,normals[x][y].y,normals[x][y].z);glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);
glTexCoord2d(0,1);glNormal3f(normals[x][y+1].x,normals[x][y+1].y,normals[x][y+1].z);glVertex3f(verts[x][y+1].x,verts[x][y+1].y,verts[x][y+1].z);
glTexCoord2d(0,0);glNormal3f(normals[x][y].x,normals[x][y].y,normals[x][y].z);glVertex3f(verts[x][y].x,verts[x][y].y,verts[x][y].z);
polies++;
}
}
glEnd();
}
}


Your method had 2 problems :
- You always pushed names without popping, thus experiencing stack overflow,
- You called glPushName between glBegin and glEnd, which is forbidden.

Edited by - vincoof on February 5, 2002 5:07:44 AM

ok... I see the problem, This puts me basically back to square one, because I cant draw my triangles like that, although I may be able to find another way. Thanks