# A peculiar rounding error

This topic is 4825 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Windows is struggling to be precise (or more specifically, the command shell in Windows XP) to 2 d.p. Here's a test code that multiplies 1105 by 5.7 (then divides by 100.0) and sets the precision to 2 d.p. However, instead of yielding 62.985 as the answer, it's storing as 62.98499... and rounding to 62.98 to 2 d.p I've tried this on two PCs so far - any comments and help appreciated (I need better precision to 2 d.p as the unit digit represents pence in financial terms)

#include <iostream>
#include <iomanip>

using namespace std;

int main()
{
int x;
double y;

x = 1105;
y = 5.700;
// Any calculator worth it's salt (or any paper/pencil arithmetic) shows
// that x*y/100 = 62.985 exactly. To 2 d.p it is 62.99

cout << setiosflags(std::ios::fixed | std::ios::showpoint)
<< setprecision(2) << x*y/100.0 << endl;

// This rounds it to 62.98 (fixed format, to 2 d.p)
// It's worth noting that to 3 d.p it correctly gives 62.985

return 0;
}



##### Share on other sites
try casting x to float or double before you do anything with it. The left hand side type is pretty significant for this kind of thing, and x is just an int
e.g.

<< setprecision(2) << ((double)x)*y/100.0 << endl;

##### Share on other sites
You will always get rounding precision errors because of the IEEE float memory respresentation..
The best you can do is to use higher precision data type, double etc....

##### Share on other sites
Yeh - I've tried all kinds of casting (that included) - even long doubles, but it still gives 62.98 as the answer. It seems to have a low precision of about 5 sig. figures.

Awful.

##### Share on other sites
That (paulecoyote's reply) won't help. x will automatically be converted to double since it is multiplied by a double. The problem is that floating point numbers cannot store most numbers exactly. The sacrifice precision for much greater range. Instead of using a floating point variable for what is actually a discrete quantity just store the number of pence as an integral value. Then to print it you can just do:
cout << (pence / 100) << '.' << width(2) << setfill('0') << (pence % 100) << endl;
EDIT: actually, rereading your post, I'm not sure if this is sufficient. I had assumed that x was a quantity and y was a price in pounds, but it clearly isn't (since you divide by 100). However the principle of fixed-point arithmetic may still be applicable.

Enigma

##### Share on other sites
Or, you could just add 0.005f to your result, before displaying it, then it would round correctly

##### Share on other sites
Quote:
 I had assumed that x was a quantity and y was a price in pounds, but it clearly isn't (since you divide by 100).

The 5.7 value is the cost in pence whilst the 1105 is the number of units (it's actually part of an energy bill calculation)- so the final value will have a fractional part of pence.

##### Share on other sites
Quote:
 Or, you could just add 0.005f to your result, before displaying it, then it would round correctly

I've just tried that and it works on 62.985. However, it's rounding up another number so that it's 1 pence too high :/

##### Share on other sites
Why is because floating point numbers are stored in base 2. Just like why you can't write 1/3 out exacly in base 10. So if you had to use limited percision decimal math floor(1/3+1/3+1/3)==0. There's "more" numbers that have this problem in base 2 then base 10 (because 5 divides 10 but not 2). For instance .1.

As for fixing this it kinda depends on what your trying to solve. Doing all the math with ints and then adding the decimal when you write could work. Or using rational numbers. If you start adding in irrationals (pi or e), then those wont work. Windows calc uses something they call extend precision (from the calc help it looks like it uses ratinals until you use an irrational number.

##### Share on other sites
Either use a higher precision floating point type or switch to a decimal representation.

• ### What is your GameDev Story?

In 2019 we are celebrating 20 years of GameDev.net! Share your GameDev Story with us.

• 28
• 16
• 10
• 10
• 11
• ### Forum Statistics

• Total Topics
634103
• Total Posts
3015539
×