Division and precision in c++
Author
I've come upon a very annoying problem while doing division in c++..
I'll demonstrate the problem with an example:
double a = double(500000000)/9;
then the value of a = 55555555.555555552
but when I do:
int b = 500000000;
int c = 9;
double a = double(b)/c;
then the value of a = 55555556.000000000
Why is there a difference when I'm doing the exact same operation? For my game I want to be able to divide variables b and c without losing precision but I don't know how.
Floating Point (especially the section "Accuracy problems")
double a = double(b)/c;
Here you divide a double with an integer, therefore the compiler decides for itself whichever of the two to convert back and forth.
In your case, it must have chosen to convert the result to an integer and then assign it to a double.
This should yield the value 55555555.555555552 (or something close to that in contrast to the FP semantics)
double a = double(b)/(double)c;
double a = double(b)/c;
Here you divide a double with an integer, therefore the compiler decides for itself whichever of the two to convert back and forth.
In your case, it must have chosen to convert the result to an integer and then assign it to a double.
This should yield the value 55555555.555555552 (or something close to that in contrast to the FP semantics)
double a = double(b)/(double)c;
There is no difference. The following program gives identical output for both values of a:
Output is:
Post your exact code and we might be able to see what's wrong.
#include <iostream>using namespace std;int main(){ double a = double(500000000)/9; cout << fixed << a << endl; int b = 500000000; int c = 9; a = double(b)/c; cout << fixed << a << endl;}Output is:
55555555.55555655555555.555556Post your exact code and we might be able to see what's wrong.
it is still truncating you to an int. You could actually need to have
double a = double(b) / double(c);
double a = double(b) / double(c);
If I'm not mistaken, only one side of / is required to be a double to make the result double (the compiler shouldn't be free to choose the result type).
From the standard:
It is surprising to see that the OP's "wrong" result has been truncated "upwards".
From the standard:
Quote:
--Otherwise, if either operand is double, the other shall be converted to double.
It is surprising to see that the OP's "wrong" result has been truncated "upwards".
Author
Yes it's true I've already tried converting both operands but I'm still getting the same result.
the exact code of my program is:
int b = 500000000;
int c = 9;
double a = double(b)/double(c);
a = double(500000000)/9;
And I check the value of variable 'a' in the debug mode of visual c++.
on line 3 , the value of a becomes 55555556.000000000
on line 4 , the value of a becomes 55555555.555555552
Does anyone have a clue?
even when I do:
double b = 500000000;
double c = 9;
double a = b/c;
the value of a is still 55555556.000000000
the exact code of my program is:
int b = 500000000;
int c = 9;
double a = double(b)/double(c);
a = double(500000000)/9;
And I check the value of variable 'a' in the debug mode of visual c++.
on line 3 , the value of a becomes 55555556.000000000
on line 4 , the value of a becomes 55555555.555555552
Does anyone have a clue?
even when I do:
double b = 500000000;
double c = 9;
double a = b/c;
the value of a is still 55555556.000000000
That exact code gives equal output for both operations (55555555.555556) when compiled with g++. The discrepancy you see might be due to a debugger quirk. Try compiling and printing the result of the calculations through cout or printf.
By the way, what version of MSVC are you using? If it's a very old version (MSVC 6), there might be issues with nonstandard behaviour.
By the way, what version of MSVC are you using? If it's a very old version (MSVC 6), there might be issues with nonstandard behaviour.
Quote:Original post by TaxCollector
And I check the value of variable 'a' in the debug mode of visual c++.
on line 3 , the value of a becomes 55555556.000000000
on line 4 , the value of a becomes 55555555.555555552
Can't replicate with MVS2008. All debug outputs show correct value.
What does debug mode mean anyway, as tooltip, as watch, compiled as debug configuration?
Author
After experimenting with the code I've found that at the very start of the program the calculation is done as expected.
It is only after calling the function
d3d->CreateDevice(D3DADAPTER_DEFAULT,
D3DDEVTYPE_HAL,
hWnd,
D3DCREATE_SOFTWARE_VERTEXPROCESSING,
&d3dpp,
&d3ddev);
(where d3ddev is an LPDIRECT3DDEVICE9;
and d3dpp is a D3DPRESENT_PARAMETERS;)
that the problem arises. So maybe this problem should've been posted in the DirectX part of the forum but I did not know the problem was caused by directX when I first posted the question. Because I'm a novice in DirectX, I do not know how to solve the issue.
It is only after calling the function
d3d->CreateDevice(D3DADAPTER_DEFAULT,
D3DDEVTYPE_HAL,
hWnd,
D3DCREATE_SOFTWARE_VERTEXPROCESSING,
&d3dpp,
&d3ddev);
(where d3ddev is an LPDIRECT3DDEVICE9;
and d3dpp is a D3DPRESENT_PARAMETERS;)
that the problem arises. So maybe this problem should've been posted in the DirectX part of the forum but I did not know the problem was caused by directX when I first posted the question. Because I'm a novice in DirectX, I do not know how to solve the issue.
I believe DirectX modifies the way the floating point processor works. There is a flag "D3DCREATE_FPU_PRESERVE" that can be used (somewhere; I don't use DirectX [smile]) to keep the original behaviour. See if this is the source of your problem.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
