C# double type... strange occurrence with precision

Started by
1 comment, last by deh0512 15 years, 10 months ago
Hi all, I've come across a strange situation with the double datatype in C#, and I thought maybe someone had an idea what could be happening. I've placed the following code in a multitude of places throughout my code (in a rather large project) with different results in different places.

double x1 = 10.0, x2 = 45.0, x3 = x1 / x2;
decimal y1 = 10.0M, y2 = 45.0M, y3 = y1 / y2;
float w1 = 10.0f, w2 = 45.0f, w3 = w1 / w2;
double z = (double)y3;

Sometimes, the outcome is: x3 = 0.22222222222222221 y3 = 0.2222222222222222222222222222 w3 = 0.222222224 z = 0.22222222222222221 This all makes perfect sense (and is correct given the relative capabilities of the datatypes). But at other points the outcome is: x3 = 0.2222222238779068 // double gets the wrong result... y3 = 0.2222222222222222222222222222 w3 = 0.222222224 z = 0.2222222238779068 // casting the decimal to double also gets the wrong result... There seems to be very little similarty among the places in my code where the error occurs. I have only seen the error in classes that use the Microsoft.DirectX and Microsoft.DirectX.Direct3D libraries, but it does not occur everywhere those libraries are used. In fact, the error occurs in some methods of a class but not in others. Also, most of the places where the error occurs are either callback methods or methods called from callback methods. Other than that I am unable to find a consistent situation that causes the faulty results. It appears to me that there is some connection between the round-off error present in the float datatype calculation and the erroneous double arithmetic (note that the float is the double value rounded to 9 decimal places... which is the maximum precision of a float... double is supposed to be accurate to at least 15 places but apparently isn't in this case). If anyone has any ideas as to how or why this could happen, please let me (and anyone else interested) know.
David Hooks
Advertisement
I don't know whether it's true for C# and which ever version of D3D you're using, but when I last used D3D9 you needed to specify D3DCREATE_FPU_PRESERVE to CreateDevice, otherwise all floating point arithmetic would be reduced to 32-bit precision.

That damn flag cost me a lot of time :)
hh10k,

Thanks man! That was exactly the problem... If I had thought it was strictly DirectX related I wouldn't have posted in this forum, but I'm glad you read it anyway.
David Hooks

This topic is closed to new replies.

Advertisement