Jump to content
  • Advertisement
Sign in to follow this  
ACCU

Weird problem [SOLVED] (I hate gcvt)

This topic is 4825 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I use VC 6 float f; f=60.1; float a=f-(long)f; result: a is 0.0999985 when using 60.1-(long)60.1 result is 0.1 How you can explain that ? [Edited by - ACCU on July 10, 2005 4:49:14 AM]

Share this post


Link to post
Share on other sites
Advertisement
Guest Anonymous Poster
rounding errors
and 60.1 is a double, 60.1f is a float.
fun fact: 0.1 cannot be exactly represented as a float/double as it is a not terminating fraction in base 2.

Share this post


Link to post
Share on other sites
that's incredible. just changed float to double and it worked. well... why for 60.1-60 flt precision is not enough that's the question. I wouldnt expect that

Share this post


Link to post
Share on other sites
Quote:
Original post by ACCU
that's incredible. just changed float to double and it worked. well... why for 60.1-60 flt precision is not enough that's the question. I wouldnt expect that


0.1 - 0.0999985 = 0.0000015

Not terribly much error there. Considering the original number is 60, that means the error is only about 0.0000025%, relative.

The problem is that numbers like 60.1, while nice in a base 10 system, dosn't directly translate into base 2. It's like trying to store the exact value of 1/3rd in base 10 decimal - you can approximate with 0.33333333, but that's still not exactly 1/3rd. In a base 3 system (digits 0,1, and 2) it's easy to store (1/3 (base-10 fractional) = 0.1 (base-3 decimal)).

You still have an error with a double, it's just that your output is truncating the result. Depending on what you're using to display your number, you may be able to manually set this. If you want this behavior with C++ iostreams using a float, just reduce the digits displayed. For example:
std::cout.setprecision( 3 );
float nearly_zero = 0.00001;
float nearly_one_nineth = 0.11111;
std::cout << nearly_zero << std::endl; //should print "0"
std::cout << nearly_one_nineth << std::endl; //should print "0.111"

Share this post


Link to post
Share on other sites
I've tried using sprintf(buf,"%.3f",f); and it works much better than gcvt.

now it is not making .09995 instead of .1
I see that because i edit my variable and show name with value in another place as well. Such behaviour was unacceptable. Now it's working correctly.
BTW, I even wrote my own conversion routine and it still worked similar .. :D Just couldn't leave like that.


Thank you all for explanation, especially MaulingMonkey. my respect.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!