Jump to content
  • Advertisement
Sign in to follow this  

GLSL problem integer and float comparison

This topic is 2945 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts


im having problem trying to figure out what happened to my code. i notice this very significant error. here is the whole code: http://pastebin.com/XERH0NNx

and i want to point out this line:

int startPointValue = (int)startPointValuev4.r;
if(startPointValue*1.0 == (19049758.0+0.5) ) color = vec4(0.0, 0.0, 1.0, 1.0);

dont mind the complex number, i just want to figure out the 0.5 part which makes it a float. and startPointValue*1.0 is a float too, but since it was an integer, it does not suppose to has a decimal value.

when i draw the glsl, it shows a blue spot on the screen which means there are a time where startPointValue*1.0 == (19049758.0+0.5). And It shouldnt, right?

Did I miss something? its a basic math, did i do anything wrong on my code?

thanks in advance

Share this post

Link to post
Share on other sites
You cannot typecast in glsl. If you want to convert a float to an int you have to use a constructor.

float f = 0.5;
int I = int(f);

float f = 0.5;
int I = (int)f;

The latter isn't legal syntax. Are you checking for warnings and errors when you compile? I don't really get what the rest of your problem is so this may or may not be your problem, just wanted to point that out.

Share this post

Link to post
Share on other sites
done that too, startPointValuev4.r is already an integer anyway, i just try to make sure by do a redundancy casting. im not sure what is the error too. no warning when compiling the GLSL.

im dealing with massive array of integer (tens of millions), and while the half of the array seems behaved correctly, the other half is not. this is one of the other messed up half that behaved weirdly.

any1 got any idea on this?

Share this post

Link to post
Share on other sites
I tweaked your code on pastebin, reposted here for posterity. I think that should work now. GLSL compiler doesn't do implicit casting like a C++ one, so be very careful - trace your types line by line if you ever encounter these errors.

#extension GL_EXT_gpu_shader4 : require
#extension GL_EXT_geometry_shader4 : enable
#extension GL_EXT_texture_integer : enable

uniform int depth;
varying float sphIndex;

uniform sampler3D tex;
uniform isampler3D texIndex;
uniform isampler2D texIndexStartPoint;

void main()
ivec2 vertCoord;

vec4 color = vec4(1.0, 1.0, 1.0, 1.0);
vec4 lastColor;

vertCoord.s = int(gl_TexCoord[0].s*300.0);
vertCoord.t = int(gl_TexCoord[0].t*300.0);

ivec4 startPointValuev4 = texelFetch(texIndexStartPoint, vertCoord, 0);
float startPointValue = float(startPointValuev4.r);

if( startPointValue*1.0 == (19049758.0+0.5) )
color = vec4(0.0, 0.0, 1.0, 1.0);

gl_FragColor = color;

Share this post

Link to post
Share on other sites
actually what i want to point out is the other way around.

startPointValuev4.r is an integer value. so it does not has decimal. but when i convert it to float, and compare it to a value with decimal, it gives me a true statement (which is from my code, i use draw blue pixel to check the result). im just confuse either this is a logic error or glsl are behaving weirdly when i put a massive interger texture. the problem is when im dealing with smaller texture, it behaves correctly.

Share this post

Link to post
Share on other sites

If my math is correct this number is too large to get that precision with floating point. With a floating point number you only get ~7 digits of decimal precision.

In your case it's likely that 19049758.0+0.5 = 19049758, as the 0.5 is too small to be added to such a large number.

Yeah just checked here: http://babbage.cs.qc.edu/IEEE-754/Decimal.html

and it shows in FP32 that 19049758.0 and 19049758.5 both evaluate to 19049758.

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!