GL_SELECT returns 0xFFFFFFFF hits

Started by
2 comments, last by Brother Bob 15 years, 1 month ago
Hello, I'm using GL_SELECT to select some mesh faces and edges. This is working in most cases, but in some cases the value returned by glRenderMode is 0xFFFFFFFF. I've done a lot of debugging on this and I can't understand why this is happening... I'm sure the values sent with gluPickMatrix are all in the viewport size range and the mesh hasn't changed... For example, this set of values works ok: viewport width : 607 viewport height : 690 x : 174.0 y : 304.0 w : 112.0 h : 138.0 And this one fails, (glRenderMode returns 0xFFFFFFFF) viewport width : 607 viewport height : 690 x : 174.0 y : 304.0 w : 112.0 h : 180.0 This is the source code:

bool ViewportMesh::FacesHitTest( wxPoint start, wxPoint end, ViewportManager * vm, std::list<unsigned int> &hits )
{
	unsigned int num_faces = m_mesh->GetNumFaces();

	GLuint * selection_buffer = new GLuint[ 4 * num_faces ];

	if ( !selection_buffer )
	{
		return false;
	}

	glSelectBuffer( num_faces, selection_buffer );
	glRenderMode( GL_SELECT );

	glInitNames();
	glPushName(-1);

	// The Size Of The Viewport. [0] Is <x>, [1] Is <y>, [2] Is <length>, [3] Is <width>
	GLint	viewport_size[4];
	glGetIntegerv( GL_VIEWPORT, viewport_size );

	double pick_x, pick_y;	
	double pick_w, pick_h;

	if ( start == end )
	{
		pick_x = 0.5 + double( start.x );
		pick_y = double( viewport_size[3] ) - 0.5 - double( start.y );
		pick_w = 1.0;
		pick_h = 1.0;
	}
	else
	{
		pick_x = 0.5 * double( start.x + end.x );
		pick_y = double( viewport_size[3] ) - 0.5 * double( start.y + end.y );
		pick_w = ScEng::MathD::FAbs( start.x - end.x ); 
		pick_h = ScEng::MathD::FAbs( start.y - end.y );
	}

	glMatrixMode(GL_PROJECTION);
	glPushMatrix();				
	glLoadIdentity();			
	gluPickMatrix( pick_x, pick_y, pick_w, pick_h, viewport_size );
	
	if ( vm->m_camera.perspective )
	{
		float y_view_fov = 2.0f * float( atan( tan(0.5*vm->m_camera.fov) * float(vm->m_camera.height) / float(vm->m_camera.width) ) );
		gluPerspective( GMT::RadiansToDegrees( y_view_fov ), vm->m_camera.aspect, vm->m_camera.clipping_near, vm->m_camera.clipping_far );
	}
	else
	{
		glOrtho( vm->m_camera.left, vm->m_camera.right, vm->m_camera.bottom, vm->m_camera.top, vm->m_camera.clipping_near, vm->m_camera.clipping_far );
	}

	glMatrixMode(GL_MODELVIEW);

	for ( unsigned int i_f = 0; i_f < m_mesh->GetNumFaces(); i_f++ )
	{
		glLoadName( i_f );

		glBegin(GL_POLYGON);

		for ( unsigned int i_v = 0; i_v < m_mesh->GetFaceDeg( i_f ); i_v++ )
		{
			float point[3];
			GMT::Point3D t_uv = m_mesh->GetVert( m_mesh->GetFaceVert( i_f, i_v ) ).v * m_tm;
			point[0] = t_uv.x;
			point[1] = t_uv.y;
			point[2] = t_uv.z;
			glVertex3f( point[0], point[1], point[2] );
		}

		glEnd();
	}

	glMatrixMode( GL_PROJECTION );
	glPopMatrix();				
	glMatrixMode( GL_MODELVIEW );

	unsigned int num_hits = glRenderMode( GL_RENDER );

	if ( num_hits == GL_INVALID_ENUM )
	{
		num_hits = 0;
	}

	if ( num_hits == GL_INVALID_OPERATION )
	{
		num_hits = 0;
	}

	if ( num_hits > 0 && num_hits < num_faces )
	{
		for ( unsigned int i_h = 0; i_h < num_hits; i_h++ )
		{
			GLuint min_z = selection_buffer[ 4 * i_h + 0 ];
			GLuint max_z = selection_buffer[ 4 * i_h + 1 ];
			GLuint q     = selection_buffer[ 4 * i_h + 2 ];
			GLuint hit   = selection_buffer[ 4 * i_h + 3 ];

			hits.push_back( hit );
		}
	}

	delete [] selection_buffer;

	if ( hits.empty() )
	{
		return false;
	}

	return true;
}

And here is the whole application source code, just in case: viewport_entities.cpp Line : 1476 Thank you in advance for your help.
CrackArthttp://www.crackart.orgOpen Source 3D Modeling and Texturing Application
Advertisement
There are at least two faults you need to solve first, the first which may or may not be the whole problem.

First, the value returned by glRenderMode is GLint but you're storing it in an unsigned int. glRenderMode can return values that cannot be stored in an unsigned integer. I leave it up to you to catch the actual return value in the correct variable type and read the documentation on what the value means.

Second, errors other than the one above are returned by glGetError, not by special return values from glRenderMode. The only time num_hits == GL_INVALID_ENUM, for example, is when you selected 1280 objects.
Hello Brother Bob,

Thank you for your help, but unfortunately I'm still having the same issue.

As GLint the value returned by glRenderMode is -1.

And glGetError returns GL_NO_ERROR.

I was just checking num_hits against GL_INVALID_ENUM and GL_INVALID_OPERATION to see if 0xFFFFFFFF was an error error, but not.

Is there any other way to perform selection with OpenGL? In any case this GL_SELECT method returns crossing selection, and I don't know if there is a way to get only the faces that are rendered completely inside the pick window.

I need also to use a non-rectangular pick window (circle, fence, free, paint).

Thank you again for your help

Diego
CrackArthttp://www.crackart.orgOpen Source 3D Modeling and Texturing Application
The correct return value is -1, as you say. Look up some documentation to see what that value means. The official API specification, for example, is available here.

Intersecting geometry selection is the only one available in OpenGL though, and its volume is limited to the shape you can form the view volume into with matrices (and possibly the few clip planes, not sure where exactly the selection is determined).

There are, however, manual selection methods. You can do your own geometrical based selection, or some color based picking (or stencil based, anything where you can control some buffer content during rendering and read it back). Color based picking allows for arbitrary shapes, but is limited to pixel resolution. Render individual objects with unique colors (or stencil values) and read back desired selection area. Compare the values with what you rendered.

This topic is closed to new replies.

Advertisement