# C# double type... strange occurrence with precision

This topic is 3671 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hi all, I've come across a strange situation with the double datatype in C#, and I thought maybe someone had an idea what could be happening. I've placed the following code in a multitude of places throughout my code (in a rather large project) with different results in different places.
double x1 = 10.0, x2 = 45.0, x3 = x1 / x2;
decimal y1 = 10.0M, y2 = 45.0M, y3 = y1 / y2;
float w1 = 10.0f, w2 = 45.0f, w3 = w1 / w2;
double z = (double)y3;


Sometimes, the outcome is: x3 = 0.22222222222222221 y3 = 0.2222222222222222222222222222 w3 = 0.222222224 z = 0.22222222222222221 This all makes perfect sense (and is correct given the relative capabilities of the datatypes). But at other points the outcome is: x3 = 0.2222222238779068 // double gets the wrong result... y3 = 0.2222222222222222222222222222 w3 = 0.222222224 z = 0.2222222238779068 // casting the decimal to double also gets the wrong result... There seems to be very little similarty among the places in my code where the error occurs. I have only seen the error in classes that use the Microsoft.DirectX and Microsoft.DirectX.Direct3D libraries, but it does not occur everywhere those libraries are used. In fact, the error occurs in some methods of a class but not in others. Also, most of the places where the error occurs are either callback methods or methods called from callback methods. Other than that I am unable to find a consistent situation that causes the faulty results. It appears to me that there is some connection between the round-off error present in the float datatype calculation and the erroneous double arithmetic (note that the float is the double value rounded to 9 decimal places... which is the maximum precision of a float... double is supposed to be accurate to at least 15 places but apparently isn't in this case). If anyone has any ideas as to how or why this could happen, please let me (and anyone else interested) know.

##### Share on other sites
I don't know whether it's true for C# and which ever version of D3D you're using, but when I last used D3D9 you needed to specify D3DCREATE_FPU_PRESERVE to CreateDevice, otherwise all floating point arithmetic would be reduced to 32-bit precision.

That damn flag cost me a lot of time :)

##### Share on other sites
hh10k,

Thanks man! That was exactly the problem... If I had thought it was strictly DirectX related I wouldn't have posted in this forum, but I'm glad you read it anyway.

1. 1
2. 2
Rutin
19
3. 3
4. 4
5. 5

• 14
• 12
• 9
• 12
• 9
• ### Forum Statistics

• Total Topics
631435
• Total Posts
3000059
×