Jump to content
  • Advertisement
Sign in to follow this  
embpos

A peculiar rounding error

This topic is 4732 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Windows is struggling to be precise (or more specifically, the command shell in Windows XP) to 2 d.p. Here's a test code that multiplies 1105 by 5.7 (then divides by 100.0) and sets the precision to 2 d.p. However, instead of yielding 62.985 as the answer, it's storing as 62.98499... and rounding to 62.98 to 2 d.p I've tried this on two PCs so far - any comments and help appreciated (I need better precision to 2 d.p as the unit digit represents pence in financial terms)

#include <iostream>
#include <iomanip>

using namespace std;

int main()
{
int x;
double y;


x = 1105;
y = 5.700;
// Any calculator worth it's salt (or any paper/pencil arithmetic) shows
// that x*y/100 = 62.985 exactly. To 2 d.p it is 62.99

cout << setiosflags(std::ios::fixed | std::ios::showpoint)
     << setprecision(2) << x*y/100.0 << endl;

// This rounds it to 62.98 (fixed format, to 2 d.p)
// It's worth noting that to 3 d.p it correctly gives 62.985


return 0;
}





Share this post


Link to post
Share on other sites
Advertisement
try casting x to float or double before you do anything with it. The left hand side type is pretty significant for this kind of thing, and x is just an int
e.g.

<< setprecision(2) << ((double)x)*y/100.0 << endl;

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
You will always get rounding precision errors because of the IEEE float memory respresentation..
The best you can do is to use higher precision data type, double etc....

Share this post


Link to post
Share on other sites
Yeh - I've tried all kinds of casting (that included) - even long doubles, but it still gives 62.98 as the answer. It seems to have a low precision of about 5 sig. figures.

Awful.

Share this post


Link to post
Share on other sites
That (paulecoyote's reply) won't help. x will automatically be converted to double since it is multiplied by a double. The problem is that floating point numbers cannot store most numbers exactly. The sacrifice precision for much greater range. Instead of using a floating point variable for what is actually a discrete quantity just store the number of pence as an integral value. Then to print it you can just do:
cout << (pence / 100) << '.' << width(2) << setfill('0') << (pence % 100) << endl;
EDIT: actually, rereading your post, I'm not sure if this is sufficient. I had assumed that x was a quantity and y was a price in pounds, but it clearly isn't (since you divide by 100). However the principle of fixed-point arithmetic may still be applicable.

Enigma

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Or, you could just add 0.005f to your result, before displaying it, then it would round correctly

Share this post


Link to post
Share on other sites
Quote:

I had assumed that x was a quantity and y was a price in pounds, but it clearly isn't (since you divide by 100).

The 5.7 value is the cost in pence whilst the 1105 is the number of units (it's actually part of an energy bill calculation)- so the final value will have a fractional part of pence.

Share this post


Link to post
Share on other sites
Quote:

Or, you could just add 0.005f to your result, before displaying it, then it would round correctly

I've just tried that and it works on 62.985. However, it's rounding up another number so that it's 1 pence too high :/

Share this post


Link to post
Share on other sites
Why is because floating point numbers are stored in base 2. Just like why you can't write 1/3 out exacly in base 10. So if you had to use limited percision decimal math floor(1/3+1/3+1/3)==0. There's "more" numbers that have this problem in base 2 then base 10 (because 5 divides 10 but not 2). For instance .1.

As for fixing this it kinda depends on what your trying to solve. Doing all the math with ints and then adding the decimal when you write could work. Or using rational numbers. If you start adding in irrationals (pi or e), then those wont work. Windows calc uses something they call extend precision (from the calc help it looks like it uses ratinals until you use an irrational number.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!