Picking problems - bad return values

Started by
7 comments, last by Icetigris 15 years, 9 months ago
I'm trying to select a cube by clicking on it using GL_SELECT render mode type selection, but it keeps returning -1 or 0 arbitrarily. I've been told to write my own picking code, but I'm not really sure how to do it. I am using jogl in CentOS 5 (RedHat). Here's my picking function:
public void pick(GL gl)
    {
        if(mouseLeftClicked)
        {
            System.out.println("you clicked your mouse and are attempting to pick something");
            IntBuffer selectionBuffer = BufferUtil.newIntBuffer(BUFSIZE);
            int hits = 0;
            IntBuffer viewport = BufferUtil.newIntBuffer(4);        
        
            gl.glGetIntegerv(GL.GL_VIEWPORT, viewport); //setup viewport
            gl.glSelectBuffer(hits, selectionBuffer); //setup hits buffer
            
            gl.glRenderMode(GL.GL_SELECT); //put gl context into selection mode
            gl.glInitNames(); //initialise name stack
            gl.glPushName(0); //push crap name onto stack to avoid error
            gl.glPopName(); //pop crap name off name stack
            
            gl.glMatrixMode(GL.GL_PROJECTION);
            gl.glPushMatrix();
            
                gl.glLoadIdentity();
                glu.gluPickMatrix(mouseX, viewport.get(3) - mouseY, pickWidth, pickHeight, viewport);
                glu.gluPerspective(45.0f, ((float)winWidth/(float)winHeight), 1.0, 50.0);        
                
                gl.glMatrixMode(GL.GL_MODELVIEW);
                
                cube.drawCubes(gl);
                
                gl.glMatrixMode(GL.GL_PROJECTION);
                
            gl.glPopMatrix();
            
            gl.glMatrixMode(GL.GL_MODELVIEW);
            
            gl.glFlush();
            hits = gl.glRenderMode(GL.GL_RENDER);
            
            processHits(hits, selectionBuffer);
            
            mouseLeftClicked = false;
            System.out.println("oh we got to the end of picking; selectionBuffer has " + selectionBuffer.capacity() + " elements");
        }
    }
Show me a sane man and I will cure him for you.
Advertisement
I actually recently posted a similar problem, but figured out that I was getting -1 because my pickbuffer size was too small.

As far as the 0 result, you might try temporarily commenting out glRenderMode calls, so basically you aren't actually doing picking, but you will get to see your picking window get drawn. This is a good way to ensure that your selection area is correct.

If your selection area looks correct and your pick buffer is sufficiently large, I would guess your problem lies in the way you do naming.
I think it might be a problem with naming, but I'm not sure. When I commented out the glRenderMode(GL_SELECT) and clicked around, colours from the wrong parts of my rainbow debug cube flashed. I'm not sure why they would do that unless it's a translation or z-buffer issue. I also don't think my pick buffer is too small, since there are 54 cubes on screen and the buffer is 512 elements in size.

The way I have it set up is that I have a class called CubeFactory which makes a list of Cube objects, each of which have their own name (i.e.
int name
). Each cube draws itself and so does glLoadName when drawn in selection mode.

Here is the draw function in Cube:
 /**  * Draws a single cube  * @param gl  */ public void draw(GL gl) {     gl.glPushMatrix();     gl.glTranslatef(location.getX(), location.getY(), location.getZ());          if(amISelected)     {//have to put a glBegin and glEnd in here         gl.glColor4f(1.0f, 1.0f, 1.0f, 1.0f);         glut.glutWireCube(size + 0.1f);                  System.out.println("You picked cube #" + name);     }          //ONLY WORKS WHEN IN GL_SELECT RENDER MODE     gl.glLoadName(name);               gl.glEnableClientState(gl.GL_VERTEX_ARRAY);     gl.glEnableClientState(gl.GL_COLOR_ARRAY);     gl.glVertexPointer(3, gl.GL_FLOAT, 0, vertexBuffer);     gl.glColorPointer(4, gl.GL_FLOAT, 0, colorBuffer);          gl.glDrawElements(gl.GL_TRIANGLES, 36, gl.GL_UNSIGNED_INT, indicies);          gl.glDisableClientState(gl.GL_COLOR_ARRAY);     gl.glDisableClientState(gl.GL_VERTEX_ARRAY);              gl.glPopMatrix(); }
Show me a sane man and I will cure him for you.
First off, the pickbuffer size can be deceiving. Each "hit" in the pickbuffer takes up more than one byte. It takes up...4 ints at the least (16 bytes), if I remember correctly. I suppose the point is it doesn't really hurt to expand the buffer and try. I ended up expanding mine up to 4096 before I got rid of the -1 return.

Something to try - instead of using LoadName, use PushName and PopName. If you never call PushName and you use LoadName, a GL error will be set because there's nothing on the name stack. I don't use LoadName at all, but rather Push and Pop names off. If you want to use LoadName, make sure you've loaded at least one name onto the stack using PushName before calling LoadName.
I tried using glPushName and glPopName inititally and when I put them back nothing changed. I expanded the pick buffer to 4096 and I'm still getting -1 returns, even with 1 cube being drawn.
Show me a sane man and I will cure him for you.
I'm looking at your code, and I think this might be a problem:

gl.glPushName(0); //push crap name onto stack to avoid error
gl.glPopName(); //pop crap name off name stack

The purpose of Pushing a "crap name" onto the stack is to avoid calling glLoadName with an empty stack. So pushing a name onto the stack and immediately popping it off doesn't do you any good. Are you polling glError somewhere to make sure there's nothing else going wrong?

If those avenues don't provide you with any answers, I have one last idea you might try. Do picking on the whole screen. First, make sure you aren't messing up the matrix calculations somewhere by commenting out glRenderMode calls and drawing the picking window, only this time your call to gluPickMatrix will look something like this:

glu.gluPickMatrix(winWidth/2, winHeight/2, winWidth, winHeight, viewport);

It will be really obvious if there's a problem with, say, your modelview matrix, because you'll see that there are two different scenes being drawn.

Once you've confirmed that the scenes being drawn are the same, go ahead and run actual picking on the entire scene, and see what your results are. If you're getting -1, don't hesitate to increase your pickbuffer size to epic proportions (32768 even).

If that doesn't work, I don't have any clue where the problem lies.
I expanded the pick buffer to the size you said, checked to see if it was drawing improperly (it is drawing fine; I checked by commenting out glRenderMode(GL_SELECT) and only drawing the scene on click), and commented out that first glPopName. I'm not sure if I'm using glGetError() right, but that returned null.

Here is my current pick function:
public void pick(GL gl)    {        if(mouseLeftClicked)        {            System.out.println("you clicked your mouse and are attempting to pick something");            IntBuffer selectionBuffer = BufferUtil.newIntBuffer(BUFSIZE);            int hits = 0;            double[] worldCoords = new double[3];            IntBuffer viewport = BufferUtil.newIntBuffer(4);              DoubleBuffer modelview = BufferUtil.newDoubleBuffer(16);            DoubleBuffer projection = BufferUtil.newDoubleBuffer(16);                        DoubleBuffer objectPosition = BufferUtil.newDoubleBuffer(3);                    gl.glGetIntegerv(GL.GL_VIEWPORT, viewport); //save viewport into buffer            gl.glGetDoublev(GL.GL_PROJECTION, projection); //save projection matrix into buffer            gl.glGetDoublev(GL.GL_PROJECTION, modelview); //save modelview matrix into buffer                        viewport.rewind();            projection.rewind();            modelview.rewind();            objectPosition.rewind();                        gl.glSelectBuffer(hits, selectionBuffer); //setup hits buffer                        //int realY = viewport.get(3) - winHeight -1;            gl.glRenderMode(GL.GL_SELECT); //put gl context into selection mode            gl.glInitNames(); //initialise name stack            gl.glPushName(0); //push crap name onto stack to avoid error            //gl.glPopName(); //pop crap name off name stack                        gl.glMatrixMode(GL.GL_PROJECTION);            gl.glPushMatrix();                            gl.glLoadIdentity();                glu.gluPickMatrix(winWidth/2, winHeight/2, winWidth, winHeight, viewport);                //glu.gluPickMatrix(mouseX, viewport.get(3) - mouseY, pickWidth, pickHeight, viewport);                glu.gluPerspective(45.0f, ((float)winWidth/(float)winHeight), 1.0, 50.0);                                gl.glMatrixMode(GL.GL_MODELVIEW);                                //cube.drawCubes(gl);                c.draw(gl);                                gl.glMatrixMode(GL.GL_PROJECTION);                            gl.glPopMatrix();                        gl.glMatrixMode(GL.GL_MODELVIEW);                        gl.glFlush();            hits = gl.glRenderMode(GL.GL_RENDER);                        //glu.gluUnProject(winWidth, realY, 10.0f, modelview, projection, viewport, objectPosition);                        processHits(hits, selectionBuffer);                        mouseLeftClicked = false;            System.out.println(gl.glGetString(gl.glGetError()));            System.out.println("oh we got to the end of picking; selectionBuffer has " + selectionBuffer.capacity() + " elements");            //System.out.println("gluUnProject results: x: " + objectPosition.get(0) + " y: " + objectPosition.get(1) + " z: " + objectPosition.get(2));        }    }        public void processHits(int hits, IntBuffer buffer)    {        cube.deselectAll();        System.out.println("in processHits; hits = " + hits);        if(hits > 0)        {            int aHit = buffer.get(3);            //gotta figure out how to select the cube with name buffer.get(3)            System.out.println("Your id: " + aHit);        }    }


Here is my current draw function:
public void draw(GL gl) {     gl.glPushMatrix();     gl.glTranslatef(location.getX(), location.getY(), location.getZ());          if(amISelected)     {//have to put a glBegin and glEnd in here         gl.glColor4f(1.0f, 1.0f, 1.0f, 1.0f);         glut.glutWireCube(size + 0.1f);                  System.out.println("You picked cube #" + name);     }          //ONLY WORKS WHEN IN GL_SELECT RENDER MODE         gl.glPushName(name);               gl.glEnableClientState(gl.GL_VERTEX_ARRAY);     gl.glEnableClientState(gl.GL_COLOR_ARRAY);     gl.glVertexPointer(3, gl.GL_FLOAT, 0, vertexBuffer);     gl.glColorPointer(4, gl.GL_FLOAT, 0, colorBuffer);          gl.glDrawElements(gl.GL_TRIANGLES, 36, gl.GL_UNSIGNED_INT, indicies);          gl.glDisableClientState(gl.GL_COLOR_ARRAY);     gl.glDisableClientState(gl.GL_VERTEX_ARRAY);             gl.glPopName();          gl.glPopMatrix(); }
Show me a sane man and I will cure him for you.
Not really sure what to tell you. You could find a simple implementation online that you know works for someone else and try using that as it is to see if you can get it to work. Then, try and figure out how they're different. I can't see anything immediately wrong with your code, but you never know.
I figured out why it was returning -1. I had is as

gl.glSelectBuffer(hits, selectionBuffer); //setup hits buffer


hits is 0 here

but it needed to be:
gl.glSelectBuffer(BUFSIZE, selectionBuffer); //setup hits buffer


That said, it's acting wacky, not returning ids properly and sometimes not returning a hit when I don't click a cube straight on (in the case of the 1 cube)
Show me a sane man and I will cure him for you.

This topic is closed to new replies.

Advertisement