float error problem shader 4

Started by
0 comments, last by Ripiz 11 years, 6 months ago
does the float can handle 1/8000 numbers and multiples of it? from 1/8000 to 8000/8000 and then plus 1.0f or -1.0f

and then adding some multiples together, the anwer is another multiple but can still handle it?

rendering to a 2D texture that is 8000x8000, I think a bug of my app is because of float error.

I just relized its not 1/8000 but 1/16000 what iam using when the texture that is 8000x8000
Advertisement
If I remember correctly float has 24 bits of precision, so it should have no problems representing 1 / 16,777,216.

However, it cannot represent all the number inbetween.
1/2 - can
1/3 - cannot
1/4 - can
1/5 - cannot
1/6 - cannot
1/7 - cannot
1/8 - can
and so on...

If you add 1.0f or -1.0f to these numbers it'll only change few last digits as decimal accuracy will decrease. It'll be around
1.cccccccww
where c - correct digit, w - incorrect.

This topic is closed to new replies.

Advertisement