I am trying to determine the 3D worldspace coordinates of the 4 corners of the viewport. Such that if I inset them a little and drew lines connecting these coordinates, I would see a frame around the window. I think what I am looking for is essentially the worldspace coordinates of the near clip plane for whatever orientation the camera is currently in.

I've tried using glUnProject, but I'm not getting the expected results. It is returning coordinates that are way larger than my scene. I've various values for the window Z coordinate, but it doesn't seem to make a difference.

Thanks, I've implemented the frustum extraction and I think I can get what I want out of it, but it looks like ti will be a lot more work.

What I don't understand is why gluUnProject is not working. If I pass 0 as the window Z coordinate, I would expect to get a point on the near clip plane. For example, in this JOGL code:


Matrix p = new Matrix();
Matrix m = new Matrix();
int[] v = new int[4];
gl.glGetFloatv(GL.GL_PROJECTION_MATRIX, p.values, 0);
gl.glGetFloatv(GL.GL_MODELVIEW_MATRIX, m.values, 0);
gl.glGetIntegerv(GL.GL_VIEWPORT, v, 0);
float[] o = new float[3];
GLU.gluUnProject(100f, 100f, 0f, m.values, 0, p.values, 0, scene.camera.viewport, 0, o, 0);

I'd expect this to give me a value close to the camera's position, instead I get wildly large values in o[0] and o[1]. It doesn't matter what I pass for win x, y.

gluProject works.

The the values member of the Matrix class is just an array of 16 floats. This is Java, so I don't have GLFloat. Here is the same code without using that class:


float[] p = new float[16];
float[] m = new float[16];
int[] v = new int[4];
gl.glGetFloatv(GL.GL_PROJECTION_MATRIX, p, 0);
gl.glGetFloatv(GL.GL_MODELVIEW_MATRIX, m, 0);
gl.glGetIntegerv(GL.GL_VIEWPORT, v, 0);
float[] o = new float[3];
GLU.gluUnProject(100f, 100f, 0f, m, 0, p, 0, scene.camera.viewport, 0, o, 0);


The results are the same. Like I said, gluProject, which also uses the projection and model view matrices, works great, so I don't think it is an issue with those values.

Looks like the JOGL version of gluUnProject is broken. I wrote my own version based on the C GLU source and it works.


private float[] unProject(float winX, float winY, float winZ,
Matrix modelView, Matrix projection,
int[] view) {
float[] in = new float[4];
Matrix m = Matrix.multiply(modelView, projection).inverse();
in[0] = (winX-(float)view[0])/(float)view[2]*2f-1f;
in[1] = (winY-(float)view[1])/(float)view[3]*2f-1f;
in[2] = 2f*winZ-1f;
in[3] = 1f;

float[] out = new float[3];
out[0]=m.values[0]*in[0]+m.values[4]*in[1]+m.values[8]*in[2]+m.values[12]*in[3];
out[1]=m.values[1]*in[0]+m.values[5]*in[1]+m.values[9]*in[2]+m.values[13]*in[3];
out[2]=m.values[2]*in[0]+m.values[6]*in[1]+m.values[10]*in[2]+m.values[14]*in[3];
//out[3]=m.values[3]*in[0]+m.values[7]*in[1]+m.values[11]*in[2]+m.values[15]*in[3]; -- don't need

return out;
}