Archived

This topic is now archived and is closed to further replies.

Roof Top Pew Wee

What causes inaccuracy of large calculations.

Recommended Posts

As I was working with my tileMap editor, I was trying to line up some objects. Being that my mouse movement is all tracked with integers, I know the minimum amount I can move an object. In my case, any number after the 6th to the right of the decimal should be 0. So .000001000 would be ok, but .000000999 although very close, should never occur. However, it was occuring, and by looking through my code, I found a line similar to this was at fault: buttonFloatVar = yVelocity * constantChange * 1000000; In this case, constantChange was -.0002f; Well, I found that if I simplified the code to this, I didn't get the inaccuracies: buttonFloatVar = yVelocity * .2f * 1000; This kept all of the calculations exact. Now, I know that I can break this equation into two parts, so that the constantChange*1000000 is calculated first, then multiplied by yVelocity, so my question is not concerning a solution. I'm more curious as to why this is happening? Does the compiler determine which type of variable it should store the values as it calculates through a line of code? Any insight would be nice. Thanks, --Vic-- The future of 2D game development: Flat Red Ball [edited by - Roof Top Pew Wee on May 1, 2003 1:11:52 AM]

Share this post


Link to post
Share on other sites
sjelkjd    171
floating point is an inexact number representation. In fact, I don''t think it''s actually possible to store 0.1 directly in a float/double, although it may be displayed correctly because of rounding.

So anyway, this is a consequence of using FP.

Share this post


Link to post
Share on other sites