Sign in to follow this  
Oralloy

GLU_TESS_COMBINE callback and linux

Recommended Posts

Oralloy    154
Hello all. I've recently been working on some GLU bindings for Ocaml (the problem is not Ocaml related) and have come across a problem with GLU tesselators under linux. I decided to do some testing of GLU tesselators (using C) in order to understand the callback system used by GLU better but I have encountered some problems. The code presented below is adapted from a redbook example. The problem is centered around the combineCallback function which is the registered callback for GLU_TESS_COMBINE. GLU_TESS_COMBINE is called whenever two edges intersect. The program crashes when combineCallback is called with: *** glibc detected *** malloc() ... (fast): <some_address> *** The ... stands in place of something that I can't remember (in windows now) and <some_address> was the address where the problem was encountered. I believe the error message occurs because glibc thinks that a buffer overrun is going to happen (which represents a security problem) and therefore ends execution. The program works as expected in windows. Does anybody have any insight into why (exactly) this occurs and what I can do to prevent it in the future? Any help appreciated.
#include <GL/glut.h>
#include <stdlib.h>
#include <stdio.h>

GLdouble star[30] = {250.0, 50.0, 0.0, 
                        1.0, 0.0, 1.0,
                     325.0, 200.0, 0.0, 
                       1.0, 1.0, 0.0,
                     400.0, 50.0, 0.0, 
                       0.0, 1.0, 1.0,
                     250.0, 150.0, 0.0, 
                        1.0, 0.0, 0.0,
                     400.0, 150.0, 0.0, 
                        0.0, 1.0, 0.0};

GLUtesselator *tobj;
GLdouble *vertex;

void display (void) {
    glClear(GL_COLOR_BUFFER_BIT);
    glColor3f(1.0, 1.0, 1.0);
  
    gluTessBeginPolygon(tobj, NULL);
        gluTessBeginContour(tobj);
            gluTessVertex(tobj, star, star);
            gluTessVertex(tobj, star + 6, star + 6);
            gluTessVertex(tobj, star + 12, star + 12);
            gluTessVertex(tobj, star + 18, star + 18);
            gluTessVertex(tobj, star + 24, star + 24);
        gluTessEndContour(tobj); 
    gluTessEndPolygon(tobj);
    
    free(vertex);
        
    glFlush();
}

void beginCallback(GLenum which)
{
   glBegin(which);
}

void errorCallback(GLenum errorCode)
{
   const GLubyte *estring;

   estring = gluErrorString(errorCode);
   fprintf(stderr, "Tessellation Error: %s\n", estring);
   exit(0);
}

void endCallback(void)
{
   glEnd();
}

void vertexCallback(GLvoid *vertex)
{
   const GLdouble *pointer;
   pointer = (GLdouble *) vertex;
   
   glColor3dv(pointer+3);
   glVertex3dv(vertex);
}


void combineCallback(GLdouble coords[3], 
                     GLdouble *vertex_data[4],
                     GLfloat weight[4], GLdouble **dataOut )
{
   int i;

   vertex = (GLdouble *) malloc(6 * sizeof(GLdouble));
   
   vertex[0] = coords[0];
   vertex[1] = coords[1];
   vertex[2] = coords[2];
   for (i = 3; i < 7; i++)
      vertex[i] = weight[0] * vertex_data[0][i] 
                  + weight[1] * vertex_data[1][i]
                  + weight[2] * vertex_data[2][i] 
                  + weight[3] * vertex_data[3][i];
   *dataOut = vertex;
}

void init (void) 
{
   glClearColor(0.0, 0.0, 0.0, 0.0);
   glShadeModel(GL_SMOOTH);    

   tobj = gluNewTess();

   gluTessCallback(tobj, GLU_TESS_VERTEX, vertexCallback);
   gluTessCallback(tobj, GLU_TESS_BEGIN, beginCallback);
   gluTessCallback(tobj, GLU_TESS_END, endCallback);
   gluTessCallback(tobj, GLU_TESS_ERROR, errorCallback);
   gluTessCallback(tobj, GLU_TESS_COMBINE, combineCallback);
}

void reshape (int w, int h)
{
   glViewport(0, 0, (GLsizei) w, (GLsizei) h);
   glMatrixMode(GL_PROJECTION);
   glLoadIdentity();
   gluOrtho2D(0.0, (GLdouble) w, 0.0, (GLdouble) h);
}

void keyboard(unsigned char key, int x, int y)
{
   switch (key) {
      case 27:
         exit(0);
         break;
   }
}

int main(int argc, char** argv)
{
   glutInit(&argc, argv);
   glutInitDisplayMode(GLUT_SINGLE | GLUT_RGB);
   glutInitWindowSize(500, 500);
   glutCreateWindow(argv[0]);
   init();
   glutDisplayFunc(display);
   glutReshapeFunc(reshape);
   glutKeyboardFunc(keyboard);
   glutMainLoop();
   gluDeleteTess(tobj);
   return 0;  
}

Share this post


Link to post
Share on other sites
kinetik_mjg    298

void combineCallback(GLdouble coords[3],
GLdouble *vertex_data[4],
GLfloat weight[4], GLdouble **dataOut )
{
int i;

vertex = (GLdouble *) malloc(6 * sizeof(GLdouble));

malloc() returns NULL on allocation failure. You need to test for this and handle it appropriately.

vertex[0] = coords[0];
vertex[1] = coords[1];
vertex[2] = coords[2];
for (i = 3; i < 7; i++)

The index (i) into vertex will be over the range 3 to 6. vertex[6] is out of the bounds of the memory you've malloc()ed.

vertex[i] = weight[0] * vertex_data[0][i]
+ weight[1] * vertex_data[1][i]
+ weight[2] * vertex_data[2][i]
+ weight[3] * vertex_data[3][i];
*dataOut = vertex;
}

Share this post


Link to post
Share on other sites
Oralloy    154
Quote:

malloc() returns NULL on allocation failure. You need to test for this and handle it appropriately.


I did check for this but malloc didn't return NULL so too simplify things I removed it from the posted code.

Quote:

The index (i) into vertex will be over the range 3 to 6. vertex[6] is out of the bounds of the memory you've malloc()ed.


Thank you kindly. This was the problem. It is a schoolboy mistake that I shouldn't have made. However, the particular function was taken (almost) verbatum from the official openGL website (mistake and all). In that respect, I feel somewhat less ashamed. For reference, here is the example program:

Opengl Site

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this