Setting a colorkey in OpenGL

Published May 08, 2005
Advertisement
I found this to be truly annoying for most of the day today. By following most of the tutorials etc. I thought I had transparency down using the alpha channel etc. etc. Only to fnid that when scaling an image the border looked like it was overlapping and the color with the supposed alpha value of 0 was showing up on the edges of stuff.

Now at 1:45am and a 2 liter of coke along with some candy bars later, I have found my solution. Ok, so maybe I am not the sharpest tool in the shed but still, the amount of info for doing 2d in OpenGL that is pure crap is amazing. Hopefully the following will help some people out.

This is the code I use to set up my OpenGL window.
GLint GL2D_SetVideoMode( GLuint width, GLuint height, GLuint bpp, GLboolean fullscreen ){	SDL_Surface *screen;	// set SDL_GL attributes	// always 32 bit and double buffered	SDL_GL_SetAttribute( SDL_GL_RED_SIZE, 8 );    SDL_GL_SetAttribute( SDL_GL_GREEN_SIZE, 8 );    SDL_GL_SetAttribute( SDL_GL_BLUE_SIZE, 8 );    SDL_GL_SetAttribute( SDL_GL_DEPTH_SIZE, 32 );    SDL_GL_SetAttribute( SDL_GL_DOUBLEBUFFER, 1 );	// initialize the display window	if( (screen = SDL_SetVideoMode(width,height,bpp,(fullscreen)?SDL_OPENGL|SDL_FULLSCREEN:SDL_OPENGL)) == NULL )	{		SDL_QuitSubSystem(SDL_INIT_VIDEO);		return GL2D_ERROR;	}		// the ever important glortho call :)	glOrtho(0,width,height,0,-1.f,1.f);	// make sure everything is ok	if(glGetError() != GL_NO_ERROR)	{		SDL_QuitSubSystem(SDL_INIT_VIDEO);		return GL2D_ERROR;	}		// enable 2d textures	glEnable(GL_TEXTURE_2D);	// set the blending mode	glEnable(GL_BLEND);	glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);		// disable depth testing	glDisable(GL_DEPTH_TEST);		// tell the world we live!	printf( "GL2D initialized: %d x %d x %dbpp\n\r",		screen->w, screen->h, screen->format->BitsPerPixel );	return 0;}


That may look wierd depending on your experience with SDL+OpenGL but it is working for me. The main things to look at are the glOrtho call and the glBlendFunc call.

The glOrtho call sets up the screen so that it works just like a 2d SDL screen with top left being 0,0 and bottom right being width,height. This was simple enough to do with the material available on glOrtho as long as you don't look for a tutorial. Just read from an OpenGL reference page about what the arguments do and it should be self explainatory.

The glBlendFunc call helps us with how our transparency works. I am not completely sure exactly what it does as opposed to the other settings, I have just found this to work for me and noticed that trying to set a colorkey without these values hasn't worked for me.

Ok, now that that is done... I have made myself a wrapper for an OpenGL texture so that it hides a few things from me that I shouldn't need to care about. Here is the GL2D_SURFACE structure I have made along with my bitmap loader:
struct GL2D_SURFACE{	GLuint	texture_id;	// the texture id of the surface	SDL_Surface *surface; // an SDL surface	GL2D_COLOR *colorkey; // the surface colorkey};GL2D_SURFACE *GL2D_LoadBMP( char *bitmap ){	SDL_Surface *temp = SDL_LoadBMP( bitmap );        // I need an alpha channel to have a color key	SDL_Surface *image = SDL_DisplayFormatAlpha( temp );	SDL_FreeSurface( temp );	GL2D_SURFACE *surface = new GL2D_SURFACE;	surface->surface = image;	surface->colorkey = NULL;        // this gives us a unique texture id to bind to        // I may be using this wrong since this gets        // called for each surface I create but I have         // loaded a few textures at once and not had any        // problems.	glGenTextures( 1, &surface->texture_id );        // still not sure why i need this here but I do	glBindTexture(GL_TEXTURE_2D, surface->texture_id);        // this actually uploads the pixeldata from system        // to video memory        // notice the BytesPerPixel and the GL_BGRA        // for some reason on my windows machine the blue        // and red components on bitmaps are switched so this        // works for me. You might need to switch to GL_RGBA        // if they are switched on yours.	glTexImage2D(GL_TEXTURE_2D, 0, surface->surface->format->BytesPerPixel, surface->surface->w, surface->surface->h, 0, GL_BGRA, GL_UNSIGNED_BYTE, surface->surface->pixels);	// using GL_NEAREST vs GL_LINEAR we don't anti alis edges on rotation or scaling	glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_NEAREST);	glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_NEAREST);	// using GL_CLAMP vs GL_WRAP so that we don't get artifacts on the edges when scaling	glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_WRAP_S,GL_CLAMP);	glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_WRAP_T,GL_CLAMP);	return surface;}


The glTexParameteri function lets me set up the way textures work. This is what was giving me grief all day and I didn't even know it. All of the tutorials I found were saying use GL_LINEAR for the MIN and MAG_FILTER settings and i think one mentioned the WRAP_S and WRAP_G things with wrap. This caused me so much grief with the colorkey thing. Moving on to which, here is how I set a colorkey on a surface now:
void GL2D_SetColorKey( GL2D_SURFACE *surface, GL2D_COLOR *colorkey ){	GL2D_COLOR pixel;	GLint x,y;	SDL_LockSurface( surface->surface );	for ( y = 0; y < (Sint16)surface->surface->h; y++ )	{		for ( x = 0; x < (Sint16)surface->surface->w; x++ )		{			GL2D_GetPixel( surface, x, y, &pixel );			// reset the old colorkey back to opaque			if ( surface->colorkey && 				 pixel.red == surface->colorkey->red &&				 pixel.green == surface->colorkey->green &&				 pixel.blue == surface->colorkey->blue &&				 pixel.alpha == 0 )			{				pixel.alpha = surface->colorkey->alpha;				GL2D_PutPixel( surface, x, y, &pixel );			}						// set the new color key			if ( colorkey &&				 pixel.red == colorkey->red &&				 pixel.green == colorkey->green &&				 pixel.blue == colorkey->blue &&				 pixel.alpha ==  colorkey->alpha )			{				pixel.alpha = 0;				GL2D_PutPixel( surface, x, y, &pixel );			}		}	}	SDL_UnlockSurface( surface->surface );	if ( colorkey )	{		if ( !surface->colorkey )			surface->colorkey = new GL2D_COLOR;		memcpy( surface->colorkey, colorkey, sizeof(surface->colorkey) );	}	else	if ( surface->colorkey )	{		delete surface->colorkey;		surface->colorkey = NULL;	}	glBindTexture(GL_TEXTURE_2D, surface->texture_id);	// using GL_NEAREST vs GL_LINEAR we don't anti alis edges on rotation or scaling	glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_NEAREST);	glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_NEAREST);	// using GL_CLAMP vs GL_WRAP so that we don't get artifacts on the edges when scaling	glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_WRAP_S,GL_CLAMP);	glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_WRAP_T,GL_CLAMP);	glTexImage2D(GL_TEXTURE_2D, 0, surface->surface->format->BytesPerPixel, surface->surface->w, surface->surface->h, 0, GL_BGRA, GL_UNSIGNED_BYTE, surface->surface->pixels);}

Pretty ugly but this was the most portable way I could figure out how to do it. Since bitmaps cannot hold an alpha channel I could just modify the bitmap loading code to set a colorkey at load time before the SDL_DisplayFormatAlpha call using SDL_SetColorkey and then the DisplayFormatAlpha call automatically does this for me. This is all well and good but what if I want to change or turn off the colorkey? I found modifying the pixels by brute force and then re-loading the pixeldata to the texture worked best for me. Also, in the future, this can be modified pretty easily to allow for multiple colorkeys which I never could get working on directdraw.

I did not add in the SDL_Image library but I am sure you could always just replace SDL_LoadBMP with SDL_LoadImage and everything should work ok but I will have to wait and see.

Ok, I am tired and need to go to bed, but I will be working more on this tomorrow.

I really need to thank PnP Bios again for his work on the hxRender library and some posts by Dave Perman and Kylotan a while back which have some informative stuff in them. I will have to go through my stuff to get the links but just needed to get a shout out. Oh, and Graveyardfilla has asked and answered some good questions regarding OpenGL and 2d on the forums as well. Hopefully when I am done this stuff will help someone so they don't have to go through what I did :)
0 likes 0 comments

Comments

Nobody has left a comment. You can be the first!
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Profile
Author
Advertisement
Advertisement