slight loss of precision
Author
I''ve got a float that can only be between 0 and 1. I want this value as an integer so I can bit shift, so I''m multiplying by 1000 (cos the float is to 3 dec places), however doing this is causing a slight loss in precision. for example:
0.364 * 1000 = 363
When doing this however : float(0.364*1000) I get 363.999695
How can I fix this??
your 0.364 is actually stored as 0.363999695
so if you cast 0.363999695 * 1000 to an int, you get 363
If you only need 3 digits after decimal point, you can do this:
float f = 0.364;
int i = int(f * 1000 + 0.5);
you can also use double s.
so if you cast 0.363999695 * 1000 to an int, you get 363
If you only need 3 digits after decimal point, you can do this:
float f = 0.364;
int i = int(f * 1000 + 0.5);
you can also use double s.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
