Jump to content
  • Advertisement
Sign in to follow this  

The magically changing float value

This topic is 4746 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm having a very weird problem. It's extremely simple to explain. I'm passing the float value 0.01f as a hard-coded parameter to a function. When I place a breakpoint in that function, the value comes up as 0.0099999998. Out of desperation to understand what is going on, I've even tried entering the number as 0.0100000000f. No difference. It's causing a very nasty alignment problem in my editor while working with transformation gizmos. Does anyone understand why this is happening? I'm using Visual Studio 7 .net with C++. Thanks

Share this post


Link to post
Share on other sites
Advertisement
The float type cannot represent all values exactly due to the way they work, using mantissas and exponents. It's a trade-off that was made to gain a large, dynamic range of values (from very, very large to very, very small.) at the sacrifice of some precision.

While it's giving you problems, in most cases having 0.01f stored as 0.0099999998 is sufficiently accurate for calculations. the double type will yield more precision, but again will not be absolutely precise in all cases.

If you need an abslute, known precision you can look into fixed-point math. Again, this will only be able to represent decimals to a certain precision, but it will have the benefit that the exact precision is known.

Share this post


Link to post
Share on other sites
I was aware of accuracy problems when storing small changes in large numbers, but I didn't know it could happen in such simple numbers.

The values are modifiers for translation/scale gizmos. I could try to align the final value as each gizmo is dragged. But I don't see how even that would be possible, since I have no number to align it to.

I'm afraid that using the translation gizmo to align seperate objects will cause problems if the neighboring vertices are not exactly equal. It would cause visible cracks in the polygons. But perhaps all of the objects will be off by the same error. I can only hope until I test it.

Thanks for your help [smile]

Share this post


Link to post
Share on other sites
Quote:
Original post by Jiia
I was aware of accuracy problems when storing small changes in large numbers, but I didn't know it could happen in such simple numbers.


But actually, it's perfectly sensible.

Quick, write out one third exactly in decimal notation. What, isn't that a simple number?

Share this post


Link to post
Share on other sites
Quote:
Original post by Zahlman
But actually, it's perfectly sensible.

Quick, write out one third exactly in decimal notation. What, isn't that a simple number?

Bah, it's a terrible flaw. I say we start over. ìÜèçìá: 1 + 1

But I'll go ahead and play. Why are you bringing up 1/3 for 0.01? One third isn't a simple number. One hundredth is.
Or at least I thought it was.

edit: Oh yeah. To clarify, by "accuracy problems", I was referring to float accuracy problems. Not math in general.

Share this post


Link to post
Share on other sites
Quote:
But I'll go ahead and play. Why are you bringing up 1/3 for 0.01? One third isn't a simple number. One hundredth is. Or at least I thought it was.


One third is a perfectly "simple number" in trinary: it is precisely 0.1.

Now, because ten is divisible by two, every rational number that has a finite binary expansion also has a finite decimal representation, but it doesn't work the opposite way. The decimal 0.1 is one of the large set of numbers that do not have an exact binary representation in mantissa-exponent form.

Share this post


Link to post
Share on other sites
Quote:
Original post by Jiia
Quote:
Original post by Zahlman
But actually, it's perfectly sensible.

Quick, write out one third exactly in decimal notation. What, isn't that a simple number?

Bah, it's a terrible flaw. I say we start over. ìÜèçìá: 1 + 1

But I'll go ahead and play. Why are you bringing up 1/3 for 0.01? One third isn't a simple number. One hundredth is.
Or at least I thought it was.

edit: Oh yeah. To clarify, by "accuracy problems", I was referring to float accuracy problems. Not math in general.

One hundredth is a "simple" number in base 10, if you define "simple" as the ability to write the number in decimal notation without large numbers of or recurring decimal digits. It is simply 0.01. One third is a "difficult" number in base 10. It is 0.3 recurring. But change to base 3 and suddenly everything is different. One third is now a "simple" number: 0.1 but one hundredth is a "difficult" number: 0.000021 recurring (3 digits recur). Computers use base 2. Every number which is "simple" in base 2 is also "simple" in base 10, but the converse is not true. A number which is "simple" in base 10 may not be "simple" in base 2. One hundredth is one such number. It is "simple" in base 10, but "difficult" in base 2: 0.0000001010001 etc.

Enigma

Share this post


Link to post
Share on other sites
Quote:
Original post by Jiia
It's causing a very nasty alignment problem in my editor while working with transformation gizmos.

This is the actual problem. The cause is not the floating-point precision (which is in fact very good). So I don't think we need a discussion about floating-point representation here (though it's still interesting to be aware of).

If I understand the alignment issue correctly, it's very similar to the Directly Mapping Texels to Pixels problem. All you have to do is offset your 2D coordinates by half a pixel.

If this is not the problem, please give us more details about it (screenshot)?

Share this post


Link to post
Share on other sites
For completeness, 0.1 (decimal) is equal to 0.0 0011 0011 0011 ... in binary, the 0011 part being repeated. Since a float only stores a few bits of manitissa, the number precisely stored is not exactly 0.1 but the 0.0999... which you got.

Now it's a very good precision, especially if all you want to do is map this to a screen coordinate. The problem is somewhere else.

Share this post


Link to post
Share on other sites
I can't really give a screenshot showing what the problem will be without a bit of work. I'm not mapping to screen coordinates. I'm positioning and orienting 3D objects in my world editor using a gizmo like that in 3D studio or such. Errors in the translation and scaling will normally not be a problem.

But if I were to build.. say, a castle wall section. I could plot that in one location, plot another right beside it, move over, plot another, and so on. I could build an entire castle wall out of 2 mesh parts. If the end vertices in one wall section do not exactly match the coordinates of the vertices in the section beside it, a visible tear will continuously jump at you as the camera is rotated around in the scene. This is how 3D rendering works. Vertices being "welded" simply means they share the exact same space. When the graphics card fills those polygons, the method it uses to fill it will exactly match the method it uses in the wall next to it if the vertices are "welded". So if the walls are aligned perfectly, there is no problem. If not, big ugly problems.

But even my GUI float-number control has it's own modifier to set the gizmo snap value. So if I use 0.01 increments to increase the gizmo snap value, the snap value will be flawed as well. Guess it's time to program that GUI float control type-in feature.

Thanks for all of the information [smile]

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!