Jump to content
  • Advertisement
Sign in to follow this  
TaxCollector

Division and precision in c++

This topic is 3697 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I've come upon a very annoying problem while doing division in c++.. I'll demonstrate the problem with an example: double a = double(500000000)/9; then the value of a = 55555555.555555552 but when I do: int b = 500000000; int c = 9; double a = double(b)/c; then the value of a = 55555556.000000000 Why is there a difference when I'm doing the exact same operation? For my game I want to be able to divide variables b and c without losing precision but I don't know how.

Share this post


Link to post
Share on other sites
Advertisement
Floating Point (especially the section "Accuracy problems")

double a = double(b)/c;

Here you divide a double with an integer, therefore the compiler decides for itself whichever of the two to convert back and forth.
In your case, it must have chosen to convert the result to an integer and then assign it to a double.

This should yield the value 55555555.555555552 (or something close to that in contrast to the FP semantics)

double a = double(b)/(double)c;

Share this post


Link to post
Share on other sites
There is no difference. The following program gives identical output for both values of a:
#include <iostream>

using namespace std;

int main()
{
double a = double(500000000)/9;
cout << fixed << a << endl;

int b = 500000000;
int c = 9;
a = double(b)/c;

cout << fixed << a << endl;
}

Output is:
55555555.555556
55555555.555556


Post your exact code and we might be able to see what's wrong.

Share this post


Link to post
Share on other sites
it is still truncating you to an int. You could actually need to have

double a = double(b) / double(c);

Share this post


Link to post
Share on other sites
If I'm not mistaken, only one side of / is required to be a double to make the result double (the compiler shouldn't be free to choose the result type).

From the standard:
Quote:

--Otherwise, if either operand is double, the other shall be converted to double.


It is surprising to see that the OP's "wrong" result has been truncated "upwards".

Share this post


Link to post
Share on other sites
Yes it's true I've already tried converting both operands but I'm still getting the same result.

the exact code of my program is:

int b = 500000000;
int c = 9;
double a = double(b)/double(c);
a = double(500000000)/9;

And I check the value of variable 'a' in the debug mode of visual c++.

on line 3 , the value of a becomes 55555556.000000000
on line 4 , the value of a becomes 55555555.555555552

Does anyone have a clue?

even when I do:

double b = 500000000;
double c = 9;
double a = b/c;

the value of a is still 55555556.000000000

Share this post


Link to post
Share on other sites
That exact code gives equal output for both operations (55555555.555556) when compiled with g++. The discrepancy you see might be due to a debugger quirk. Try compiling and printing the result of the calculations through cout or printf.

By the way, what version of MSVC are you using? If it's a very old version (MSVC 6), there might be issues with nonstandard behaviour.

Share this post


Link to post
Share on other sites
Quote:
Original post by TaxCollector

And I check the value of variable 'a' in the debug mode of visual c++.

on line 3 , the value of a becomes 55555556.000000000
on line 4 , the value of a becomes 55555555.555555552


Can't replicate with MVS2008. All debug outputs show correct value.

What does debug mode mean anyway, as tooltip, as watch, compiled as debug configuration?

Share this post


Link to post
Share on other sites
After experimenting with the code I've found that at the very start of the program the calculation is done as expected.

It is only after calling the function

d3d->CreateDevice(D3DADAPTER_DEFAULT,
D3DDEVTYPE_HAL,
hWnd,
D3DCREATE_SOFTWARE_VERTEXPROCESSING,
&d3dpp,
&d3ddev);

(where d3ddev is an LPDIRECT3DDEVICE9;
and d3dpp is a D3DPRESENT_PARAMETERS;)

that the problem arises. So maybe this problem should've been posted in the DirectX part of the forum but I did not know the problem was caused by directX when I first posted the question. Because I'm a novice in DirectX, I do not know how to solve the issue.

Share this post


Link to post
Share on other sites
I believe DirectX modifies the way the floating point processor works. There is a flag "D3DCREATE_FPU_PRESERVE" that can be used (somewhere; I don't use DirectX [smile]) to keep the original behaviour. See if this is the source of your problem.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!