GLSL NaN problem

Started by
1 comment, last by RPTD 15 years, 10 months ago
I just got hit by this recently when I improved an incorrect down-sampling in my tone-mapper. Suddenly at times the screen gets totally black and never recovers from it. I track this down to the tone mapper receiving an NaN value as the average scene key. This I tracked down to be colors with all components set to NaN at the beginning of the down sample stage. Hence whenever the GLSL shaders during rendering spit out even one NaN value the entire down sampling is doomed. Now comes though the funny part: How in gods name can an nVidia card spit out a color with all NaN?. I need to get this fixed so I can at last go on and worry about this driver problem later on. Is there a way I can detect an NaN value in the shader and replace it with something like vec4(0) or such a thing to avoid a total black-out?

Life's like a Hydra... cut off one problem just to have two more popping out.
Leader and Coder: Project Epsylon | Drag[en]gine Game Engine

Advertisement
Quote:Original post by RPTDHow in gods name can an nVidia card spit out a color with all NaN?

http://steve.hollasch.net/cgindex/coding/ieeefloat.html
Most likely you're dividing by 0 or something silly like that.

As for detecting, the standard way I know of is to compare it with itself and see if it evaluates to false.
According to the IEEE standard, NaN != NaN.

But GPU's generally don't implement the complete IEEE standard, so I don't know if that'll work in your case.

But you're probably better off finding out why you end up with NaN's in the first place, than trying to hack around it like this.
Looks like an nVidia driver bug. I tracked the NaN down to some shader being executed and this tracks down to some normal being calculated beforehand. I did all the precautions like testing for 0-length before normalizing and min(*,65504) to prevent overflow but yet there creep in a handful ( up to 10 ) of NaN values during calculation.

Anybody has experience with bugged normalize call in GLSL on nVidia machines? What version should be safe of this driver bug?

Life's like a Hydra... cut off one problem just to have two more popping out.
Leader and Coder: Project Epsylon | Drag[en]gine Game Engine

This topic is closed to new replies.

Advertisement