Jump to content
  • Advertisement


This topic is now archived and is closed to further replies.


slight loss of precision

This topic is 5559 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I''ve got a float that can only be between 0 and 1. I want this value as an integer so I can bit shift, so I''m multiplying by 1000 (cos the float is to 3 dec places), however doing this is causing a slight loss in precision. for example: 0.364 * 1000 = 363 When doing this however : float(0.364*1000) I get 363.999695 How can I fix this??

Share this post

Link to post
Share on other sites
your 0.364 is actually stored as 0.363999695

so if you cast 0.363999695 * 1000 to an int, you get 363

If you only need 3 digits after decimal point, you can do this:

float f = 0.364;
int i = int(f * 1000 + 0.5);

you can also use double s.

Share this post

Link to post
Share on other sites

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!