Jump to content

  • Log In with Google      Sign In   
  • Create Account

We're offering banner ads on our site from just $5!

1. Details HERE. 2. GDNet+ Subscriptions HERE. 3. Ad upload HERE.


float error problem shader 4


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
1 reply to this topic

#1 lomateron   Members   -  Reputation: 363

Like
0Likes
Like

Posted 19 October 2012 - 01:21 PM

does the float can handle 1/8000 numbers and multiples of it? from 1/8000 to 8000/8000 and then plus 1.0f or -1.0f

and then adding some multiples together, the anwer is another multiple but can still handle it?

rendering to a 2D texture that is 8000x8000, I think a bug of my app is because of float error.

I just relized its not 1/8000 but 1/16000 what iam using when the texture that is 8000x8000

Edited by lomateron, 19 October 2012 - 01:35 PM.


Sponsor:

#2 Ripiz   Members   -  Reputation: 529

Like
0Likes
Like

Posted 20 October 2012 - 02:13 AM

If I remember correctly float has 24 bits of precision, so it should have no problems representing 1 / 16,777,216.

However, it cannot represent all the number inbetween.
1/2 - can
1/3 - cannot
1/4 - can
1/5 - cannot
1/6 - cannot
1/7 - cannot
1/8 - can
and so on...

If you add 1.0f or -1.0f to these numbers it'll only change few last digits as decimal accuracy will decrease. It'll be around
1.cccccccww
where c - correct digit, w - incorrect.




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS