I'm using microsoft visual studio to do some 3D math libraries, and I spent the better part of the night tracking down a wierd bug. Basically, my compiler maintains that casting the result of a complex sqrt() to float makes it != to itself. In other words, if a function to take the magnitude of a vector returns a float, wierd shit happens. Check out this example code:
Under microsoft visual studio .net, it prints: assigned mag(a) to float c and d c and d are equal mag(a) does not equal mag(a) mag2(a) equals mag2(a) In other words, calling the same function on the same data twice does not yield equal results, if the function returns a float. An identical function which returns a double does not experience this problem. Is this a bug in the compiler or library? Am I just doing something stupid? The only possibility that I could see is that there is some kind of safety mechanism which notices when data is lost (e.g. a double is converted to a float) and sets the two pieces of data to be not equal, in case we had a situation like 1.00000...1 and 1.000000....2, where .... is more precision than a float can handle, and less than a double. [edited by - Ishan on April 28, 2003 3:44:22 AM]

does your compiler give any warnings?

I know that doubles have what, twice as much precision as floats? There''s always problems with comparing floats with ==. Beware. There is a lot of error in floating point precision.

Yes, you shouldn't ever compare floats or doubles with == for equality.

Instead, do:

bool IsEqual(float lhs,float rhs,float tolerance=1e-4){ return fabs(lhs-rhs) < tolerance;}  

Note: VS 2003 finally has non-retarded abs,sin,cos,sqrt routines!

[edited by - sjelkjd on April 28, 2003 4:03:09 AM]

quote:Original post by sjelkjd
Yes, you shouldn''t ever compare floats or doubles with == for equality.

Instead, do:

bool IsEqual(float lhs,float rhs,float tolerance=1e-4){ return fabs(lhs-rhs) < tolerance;}   

Note: VS 2003 finally has non-retarded abs,sin,cos,sqrt routines!

[edited by - sjelkjd on April 28, 2003 4:03:09 AM]

Very interesting, thanks.

quote:Original post by petewood
does your compiler give any warnings?

Yeah, it says "conversion from double to float, possible loss of data"...but shouldn''t it be the same data lost every time?

This isn''t a bug as such. This is more to do with the way floating point values are handled by the CPU. On the Intel chips, the values internally are stored in double format by default.
In this program, the compiler needs to create a temporary variable to store the result of the first "mag (a)" in the line "if(mag(a) == mag(a))", this temporary is of type float. The compiler generates this code sequence:

push offset acall magstore floating point value at top of fp stack to temporary variable 

The result of the function call is returned on the top of the floating point stack - so it''s *always* a double, even if you declare the function a float. The conversion to float occurs when the returned value is saved to a variable.
The next call to "mag (a)" is the same as the first but the result is never saved to memory - it doesn''t need to be.
The comparison then becomes a compare between the temporary variable and the value on the top of the floating point stack - effectively a comparison between a float (the temporary) and a double (the fp stack) which will yield a non-equal result in this instance as the float version has lost some precision.

The first comparison works because both values have been truncated when saved to the variables, the third works because the temporary value is a double and so no precision is lost.

To solve this, either save the returned values to variables before the comparison (slower - two dwords are written) or cast the first call to a double (not quite as slow - a qword is written instead of a dword):

if((double)mag(a) == mag(a)) 

This forces the temporary value to be a double and not a float. Casting to float doesn''t work (it''s ignored).

Saving the result of the first call on the FP stack is not an option because the compiler can''t garantee that the function ''mag'' doesn''t use all the FP stack and hence lose the result of the first call (in this case it probably doesn''t though).

Skizz

This explains why I couldn''t find my last big "bug". :D
Interesting post indeed...

"He who hasn''t hacked assembly language as a youth has no heart. He who does as an adult has no brain."

quote:Original post by Skizz
This isn''t a bug as such. This is more to do with the way floating point values are handled by the CPU. On the Intel chips, the values internally are stored in double format by default.

Actually, it''s a little more complicated than that. x87 uses 80 bits of internal precision. If you use SSE, you only get 32 bits though(or 64 if you use double precision). That might sound weird, but remember that intel is trying to get everyone to use SSE2 instead of x87(even for scalar ops), since the p4 x87 is so weak.