Jump to content
  • Advertisement

Archived

This topic is now archived and is closed to further replies.

RavNaz

slight loss of precision

This topic is 5478 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I''ve got a float that can only be between 0 and 1. I want this value as an integer so I can bit shift, so I''m multiplying by 1000 (cos the float is to 3 dec places), however doing this is causing a slight loss in precision. for example: 0.364 * 1000 = 363 When doing this however : float(0.364*1000) I get 363.999695 How can I fix this??

Share this post


Link to post
Share on other sites
Advertisement
your 0.364 is actually stored as 0.363999695

so if you cast 0.363999695 * 1000 to an int, you get 363

If you only need 3 digits after decimal point, you can do this:

float f = 0.364;
int i = int(f * 1000 + 0.5);

you can also use double s.

Share this post


Link to post
Share on other sites

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!