🔒 Locked

GLSL problem integer and float comparison

Started by svnstrk Sep 26, 2010 at 8:48 PM 4 replies 18.9k views

Original Post

Sep 26, 2010 at 8:48 PM

hi,

im having problem trying to figure out what happened to my code. i notice this very significant error. here is the whole code: http://pastebin.com/XERH0NNx

and i want to point out this line:

int startPointValue = (int)startPointValuev4.r;
if(startPointValue*1.0 == (19049758.0+0.5) ) color = vec4(0.0, 0.0, 1.0, 1.0);

dont mind the complex number, i just want to figure out the 0.5 part which makes it a float. and startPointValue*1.0 is a float too, but since it was an integer, it does not suppose to has a decimal value.

when i draw the glsl, it shows a blue spot on the screen which means there are a time where startPointValue*1.0 == (19049758.0+0.5). And It shouldnt, right?

Did I miss something? its a basic math, did i do anything wrong on my code?

thanks in advance

karwosts

Sep 26, 2010 at 8:57 PM

You cannot typecast in glsl. If you want to convert a float to an int you have to use a constructor.

Good:
float f = 0.5;
int I = int(f);

Bad:
float f = 0.5;
int I = (int)f;

The latter isn't legal syntax. Are you checking for warnings and errors when you compile? I don't really get what the rest of your problem is so this may or may not be your problem, just wanted to point that out.

svnstrk

Sep 26, 2010 at 9:02 PM

done that too, startPointValuev4.r is already an integer anyway, i just try to make sure by do a redundancy casting. im not sure what is the error too. no warning when compiling the GLSL.

im dealing with massive array of integer (tens of millions), and while the half of the array seems behaved correctly, the other half is not. this is one of the other messed up half that behaved weirdly.

any1 got any idea on this?

silvermace

Sep 26, 2010 at 9:29 PM

I tweaked your code on pastebin, reposted here for posterity. I think that should work now. GLSL compiler doesn't do implicit casting like a C++ one, so be very careful - trace your types line by line if you ever encounter these errors.

#extension GL_EXT_gpu_shader4 : require#extension GL_EXT_geometry_shader4 : enable#extension GL_EXT_texture_integer : enableuniform int depth;varying float sphIndex;uniform sampler3D tex;uniform isampler3D texIndex;uniform isampler2D texIndexStartPoint;void main(){	ivec2 vertCoord;    	vec4 color = vec4(1.0, 1.0, 1.0, 1.0);	vec4 lastColor;	vertCoord.s = int(gl_TexCoord[0].s*300.0);	vertCoord.t = int(gl_TexCoord[0].t*300.0);	ivec4 startPointValuev4 = texelFetch(texIndexStartPoint, vertCoord, 0);    	float startPointValue = float(startPointValuev4.r);    	if( startPointValue*1.0 == (19049758.0+0.5) )		color = vec4(0.0, 0.0, 1.0, 1.0);    	gl_FragColor = color;}

svnstrk

Sep 26, 2010 at 9:36 PM

actually what i want to point out is the other way around.

startPointValuev4.r is an integer value. so it does not has decimal. but when i convert it to float, and compare it to a value with decimal, it gives me a true statement (which is from my code, i use draw blue pixel to check the result). im just confuse either this is a logic error or glsl are behaving weirdly when i put a massive interger texture. the problem is when im dealing with smaller texture, it behaves correctly.

karwosts

Sep 26, 2010 at 10:21 PM

Quote:
19049758.0+0.5

If my math is correct this number is too large to get that precision with floating point. With a floating point number you only get ~7 digits of decimal precision.

In your case it's likely that 19049758.0+0.5 = 19049758, as the 0.5 is too small to be added to such a large number.

Yeah just checked here: http://babbage.cs.qc.edu/IEEE-754/Decimal.html

and it shows in FP32 that 19049758.0 and 19049758.5 both evaluate to 19049758.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.