Jump to content
  • Advertisement
Sign in to follow this  
deh0512

C# double type... strange occurrence with precision

This topic is 3671 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi all, I've come across a strange situation with the double datatype in C#, and I thought maybe someone had an idea what could be happening. I've placed the following code in a multitude of places throughout my code (in a rather large project) with different results in different places.
double x1 = 10.0, x2 = 45.0, x3 = x1 / x2;
decimal y1 = 10.0M, y2 = 45.0M, y3 = y1 / y2;
float w1 = 10.0f, w2 = 45.0f, w3 = w1 / w2;
double z = (double)y3;

Sometimes, the outcome is: x3 = 0.22222222222222221 y3 = 0.2222222222222222222222222222 w3 = 0.222222224 z = 0.22222222222222221 This all makes perfect sense (and is correct given the relative capabilities of the datatypes). But at other points the outcome is: x3 = 0.2222222238779068 // double gets the wrong result... y3 = 0.2222222222222222222222222222 w3 = 0.222222224 z = 0.2222222238779068 // casting the decimal to double also gets the wrong result... There seems to be very little similarty among the places in my code where the error occurs. I have only seen the error in classes that use the Microsoft.DirectX and Microsoft.DirectX.Direct3D libraries, but it does not occur everywhere those libraries are used. In fact, the error occurs in some methods of a class but not in others. Also, most of the places where the error occurs are either callback methods or methods called from callback methods. Other than that I am unable to find a consistent situation that causes the faulty results. It appears to me that there is some connection between the round-off error present in the float datatype calculation and the erroneous double arithmetic (note that the float is the double value rounded to 9 decimal places... which is the maximum precision of a float... double is supposed to be accurate to at least 15 places but apparently isn't in this case). If anyone has any ideas as to how or why this could happen, please let me (and anyone else interested) know.

Share this post


Link to post
Share on other sites
Advertisement
I don't know whether it's true for C# and which ever version of D3D you're using, but when I last used D3D9 you needed to specify D3DCREATE_FPU_PRESERVE to CreateDevice, otherwise all floating point arithmetic would be reduced to 32-bit precision.

That damn flag cost me a lot of time :)

Share this post


Link to post
Share on other sites
hh10k,

Thanks man! That was exactly the problem... If I had thought it was strictly DirectX related I wouldn't have posted in this forum, but I'm glad you read it anyway.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!