Sign in to follow this  

GLU_TESS_COMBINE callback and linux

This topic is 4549 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello all. I've recently been working on some GLU bindings for Ocaml (the problem is not Ocaml related) and have come across a problem with GLU tesselators under linux. I decided to do some testing of GLU tesselators (using C) in order to understand the callback system used by GLU better but I have encountered some problems. The code presented below is adapted from a redbook example. The problem is centered around the combineCallback function which is the registered callback for GLU_TESS_COMBINE. GLU_TESS_COMBINE is called whenever two edges intersect. The program crashes when combineCallback is called with: *** glibc detected *** malloc() ... (fast): <some_address> *** The ... stands in place of something that I can't remember (in windows now) and <some_address> was the address where the problem was encountered. I believe the error message occurs because glibc thinks that a buffer overrun is going to happen (which represents a security problem) and therefore ends execution. The program works as expected in windows. Does anybody have any insight into why (exactly) this occurs and what I can do to prevent it in the future? Any help appreciated.
#include <GL/glut.h>
#include <stdlib.h>
#include <stdio.h>

GLdouble star[30] = {250.0, 50.0, 0.0, 
                        1.0, 0.0, 1.0,
                     325.0, 200.0, 0.0, 
                       1.0, 1.0, 0.0,
                     400.0, 50.0, 0.0, 
                       0.0, 1.0, 1.0,
                     250.0, 150.0, 0.0, 
                        1.0, 0.0, 0.0,
                     400.0, 150.0, 0.0, 
                        0.0, 1.0, 0.0};

GLUtesselator *tobj;
GLdouble *vertex;

void display (void) {
    glClear(GL_COLOR_BUFFER_BIT);
    glColor3f(1.0, 1.0, 1.0);
  
    gluTessBeginPolygon(tobj, NULL);
        gluTessBeginContour(tobj);
            gluTessVertex(tobj, star, star);
            gluTessVertex(tobj, star + 6, star + 6);
            gluTessVertex(tobj, star + 12, star + 12);
            gluTessVertex(tobj, star + 18, star + 18);
            gluTessVertex(tobj, star + 24, star + 24);
        gluTessEndContour(tobj); 
    gluTessEndPolygon(tobj);
    
    free(vertex);
        
    glFlush();
}

void beginCallback(GLenum which)
{
   glBegin(which);
}

void errorCallback(GLenum errorCode)
{
   const GLubyte *estring;

   estring = gluErrorString(errorCode);
   fprintf(stderr, "Tessellation Error: %s\n", estring);
   exit(0);
}

void endCallback(void)
{
   glEnd();
}

void vertexCallback(GLvoid *vertex)
{
   const GLdouble *pointer;
   pointer = (GLdouble *) vertex;
   
   glColor3dv(pointer+3);
   glVertex3dv(vertex);
}


void combineCallback(GLdouble coords[3], 
                     GLdouble *vertex_data[4],
                     GLfloat weight[4], GLdouble **dataOut )
{
   int i;

   vertex = (GLdouble *) malloc(6 * sizeof(GLdouble));
   
   vertex[0] = coords[0];
   vertex[1] = coords[1];
   vertex[2] = coords[2];
   for (i = 3; i < 7; i++)
      vertex[i] = weight[0] * vertex_data[0][i] 
                  + weight[1] * vertex_data[1][i]
                  + weight[2] * vertex_data[2][i] 
                  + weight[3] * vertex_data[3][i];
   *dataOut = vertex;
}

void init (void) 
{
   glClearColor(0.0, 0.0, 0.0, 0.0);
   glShadeModel(GL_SMOOTH);    

   tobj = gluNewTess();

   gluTessCallback(tobj, GLU_TESS_VERTEX, vertexCallback);
   gluTessCallback(tobj, GLU_TESS_BEGIN, beginCallback);
   gluTessCallback(tobj, GLU_TESS_END, endCallback);
   gluTessCallback(tobj, GLU_TESS_ERROR, errorCallback);
   gluTessCallback(tobj, GLU_TESS_COMBINE, combineCallback);
}

void reshape (int w, int h)
{
   glViewport(0, 0, (GLsizei) w, (GLsizei) h);
   glMatrixMode(GL_PROJECTION);
   glLoadIdentity();
   gluOrtho2D(0.0, (GLdouble) w, 0.0, (GLdouble) h);
}

void keyboard(unsigned char key, int x, int y)
{
   switch (key) {
      case 27:
         exit(0);
         break;
   }
}

int main(int argc, char** argv)
{
   glutInit(&argc, argv);
   glutInitDisplayMode(GLUT_SINGLE | GLUT_RGB);
   glutInitWindowSize(500, 500);
   glutCreateWindow(argv[0]);
   init();
   glutDisplayFunc(display);
   glutReshapeFunc(reshape);
   glutKeyboardFunc(keyboard);
   glutMainLoop();
   gluDeleteTess(tobj);
   return 0;  
}

Share this post


Link to post
Share on other sites

void combineCallback(GLdouble coords[3],
GLdouble *vertex_data[4],
GLfloat weight[4], GLdouble **dataOut )
{
int i;

vertex = (GLdouble *) malloc(6 * sizeof(GLdouble));

malloc() returns NULL on allocation failure. You need to test for this and handle it appropriately.

vertex[0] = coords[0];
vertex[1] = coords[1];
vertex[2] = coords[2];
for (i = 3; i < 7; i++)

The index (i) into vertex will be over the range 3 to 6. vertex[6] is out of the bounds of the memory you've malloc()ed.

vertex[i] = weight[0] * vertex_data[0][i]
+ weight[1] * vertex_data[1][i]
+ weight[2] * vertex_data[2][i]
+ weight[3] * vertex_data[3][i];
*dataOut = vertex;
}

Share this post


Link to post
Share on other sites
Quote:

malloc() returns NULL on allocation failure. You need to test for this and handle it appropriately.


I did check for this but malloc didn't return NULL so too simplify things I removed it from the posted code.

Quote:

The index (i) into vertex will be over the range 3 to 6. vertex[6] is out of the bounds of the memory you've malloc()ed.


Thank you kindly. This was the problem. It is a schoolboy mistake that I shouldn't have made. However, the particular function was taken (almost) verbatum from the official openGL website (mistake and all). In that respect, I feel somewhat less ashamed. For reference, here is the example program:

Opengl Site

Share this post


Link to post
Share on other sites

This topic is 4549 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this