Jump to content
  • Advertisement
Sign in to follow this  
RPTD

GLSL NaN problem

This topic is 3798 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I just got hit by this recently when I improved an incorrect down-sampling in my tone-mapper. Suddenly at times the screen gets totally black and never recovers from it. I track this down to the tone mapper receiving an NaN value as the average scene key. This I tracked down to be colors with all components set to NaN at the beginning of the down sample stage. Hence whenever the GLSL shaders during rendering spit out even one NaN value the entire down sampling is doomed. Now comes though the funny part: How in gods name can an nVidia card spit out a color with all NaN?. I need to get this fixed so I can at last go on and worry about this driver problem later on. Is there a way I can detect an NaN value in the shader and replace it with something like vec4(0) or such a thing to avoid a total black-out?

Share this post


Link to post
Share on other sites
Advertisement
Quote:
Original post by RPTDHow in gods name can an nVidia card spit out a color with all NaN?

http://steve.hollasch.net/cgindex/coding/ieeefloat.html
Most likely you're dividing by 0 or something silly like that.

As for detecting, the standard way I know of is to compare it with itself and see if it evaluates to false.
According to the IEEE standard, NaN != NaN.

But GPU's generally don't implement the complete IEEE standard, so I don't know if that'll work in your case.

But you're probably better off finding out why you end up with NaN's in the first place, than trying to hack around it like this.

Share this post


Link to post
Share on other sites
Looks like an nVidia driver bug. I tracked the NaN down to some shader being executed and this tracks down to some normal being calculated beforehand. I did all the precautions like testing for 0-length before normalizing and min(*,65504) to prevent overflow but yet there creep in a handful ( up to 10 ) of NaN values during calculation.

Anybody has experience with bugged normalize call in GLSL on nVidia machines? What version should be safe of this driver bug?

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!