• ### What is your GameDev Story?

#### Archived

This topic is now archived and is closed to further replies.

# possible bug in ms visual studio?

This topic is 5745 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I'm using microsoft visual studio to do some 3D math libraries, and I spent the better part of the night tracking down a wierd bug. Basically, my compiler maintains that casting the result of a complex sqrt() to float makes it != to itself. In other words, if a function to take the magnitude of a vector returns a float, wierd shit happens. Check out this example code:

#include <iostream>
#include <cmath>
using namespace std;

float a[3] = {1,2,3};

float mag(float * ptr){
return sqrt(ptr[0] * ptr[0] + ptr[1] * ptr[1] + ptr[2] * ptr[2]);
}

double mag2(float * ptr){
return sqrt(ptr[0] * ptr[0] + ptr[1] * ptr[1] + ptr[2] * ptr[2]);
}

void main(){
float c = mag(a);
float d = mag(a);
cout << "assigned mag(a) to float c and d" << endl;
if(c == d){
cout << "c and d are equal" << endl;
} else {
cout << "c and d are not equal" << endl;
}
if(mag(a) == mag(a)){
cout << "mag(a) equals mag(a)" << endl;
} else {
cout << "mag(a) does not equal mag(a)" << endl;
}
if(mag2(a) == mag2(a)){
cout << "mag2(a) equals mag2(a)" << endl;
} else {
cout << "mag2(a) does not equal mag2(a)" << endl;
}
}

Under microsoft visual studio .net, it prints: assigned mag(a) to float c and d c and d are equal mag(a) does not equal mag(a) mag2(a) equals mag2(a) In other words, calling the same function on the same data twice does not yield equal results, if the function returns a float. An identical function which returns a double does not experience this problem. Is this a bug in the compiler or library? Am I just doing something stupid? The only possibility that I could see is that there is some kind of safety mechanism which notices when data is lost (e.g. a double is converted to a float) and sets the two pieces of data to be not equal, in case we had a situation like 1.00000...1 and 1.000000....2, where .... is more precision than a float can handle, and less than a double. [edited by - Ishan on April 28, 2003 3:44:22 AM]

##### Share on other sites
does your compiler give any warnings?

##### Share on other sites
I know that doubles have what, twice as much precision as floats? There''s always problems with comparing floats with ==. Beware. There is a lot of error in floating point precision.

##### Share on other sites
Yes, you shouldn't ever compare floats or doubles with == for equality.

bool IsEqual(float lhs,float rhs,float tolerance=1e-4){ return fabs(lhs-rhs) < tolerance;}

Note: VS 2003 finally has non-retarded abs,sin,cos,sqrt routines!

[edited by - sjelkjd on April 28, 2003 4:03:09 AM]

##### Share on other sites
quote:
Original post by sjelkjd
Yes, you shouldn''t ever compare floats or doubles with == for equality.

bool IsEqual(float lhs,float rhs,float tolerance=1e-4){ return fabs(lhs-rhs) < tolerance;}

Note: VS 2003 finally has non-retarded abs,sin,cos,sqrt routines!

[edited by - sjelkjd on April 28, 2003 4:03:09 AM]

Very interesting, thanks.

##### Share on other sites
quote:
Original post by petewood
does your compiler give any warnings?

Yeah, it says "conversion from double to float, possible loss of data"...but shouldn''t it be the same data lost every time?

##### Share on other sites
This isn''t a bug as such. This is more to do with the way floating point values are handled by the CPU. On the Intel chips, the values internally are stored in double format by default.
In this program, the compiler needs to create a temporary variable to store the result of the first "mag (a)" in the line "if(mag(a) == mag(a))", this temporary is of type float. The compiler generates this code sequence:
push offset acall magstore floating point value at top of fp stack to temporary variable

The result of the function call is returned on the top of the floating point stack - so it''s *always* a double, even if you declare the function a float. The conversion to float occurs when the returned value is saved to a variable.
The next call to "mag (a)" is the same as the first but the result is never saved to memory - it doesn''t need to be.
The comparison then becomes a compare between the temporary variable and the value on the top of the floating point stack - effectively a comparison between a float (the temporary) and a double (the fp stack) which will yield a non-equal result in this instance as the float version has lost some precision.

The first comparison works because both values have been truncated when saved to the variables, the third works because the temporary value is a double and so no precision is lost.

To solve this, either save the returned values to variables before the comparison (slower - two dwords are written) or cast the first call to a double (not quite as slow - a qword is written instead of a dword):
if((double)mag(a) == mag(a))

This forces the temporary value to be a double and not a float. Casting to float doesn''t work (it''s ignored).

Saving the result of the first call on the FP stack is not an option because the compiler can''t garantee that the function ''mag'' doesn''t use all the FP stack and hence lose the result of the first call (in this case it probably doesn''t though).

Skizz

##### Share on other sites
This explains why I couldn''t find my last big "bug". :D
Interesting post indeed...

"He who hasn''t hacked assembly language as a youth has no heart. He who does as an adult has no brain."

##### Share on other sites
quote:
Original post by Skizz
This isn''t a bug as such. This is more to do with the way floating point values are handled by the CPU. On the Intel chips, the values internally are stored in double format by default.

Actually, it''s a little more complicated than that. x87 uses 80 bits of internal precision. If you use SSE, you only get 32 bits though(or 64 if you use double precision). That might sound weird, but remember that intel is trying to get everyone to use SSE2 instead of x87(even for scalar ops), since the p4 x87 is so weak.

##### Share on other sites
FLT_EPSILON is defined limits.h (1.192092896E-07).

so comparing the difference between to floats to that should be pretty safe on whatever your working with.

[My site|SGI STL|Bjarne FAQ|C++ FAQ Lite|MSDN|Jargon]
Ripped off from various people

• ### What is your GameDev Story?

In 2019 we are celebrating 20 years of GameDev.net! Share your GameDev Story with us.

• 27
• 16
• 10
• 10
• 11
• ### Forum Statistics

• Total Topics
634101
• Total Posts
3015528
×