Jump to content
  • Advertisement
Sign in to follow this  
lomateron

help with result of line of code

This topic is 1757 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

In HLSL 4.0 i have this line that is imposible for me to get the debugger to it(apart from having to wait around 15 minutes to build the solution in debug mode)

 

a=30030.0f;

b=11.0f;

c=true;

 

result = float(frac(a/b)==0.0f && c)

 

I expect  "result" to be 1.0f but I think "result" ends with value 0.0f, does anyone know why it ends with 0.0f, or is just paranoia? 

Share this post


Link to post
Share on other sites
Advertisement

I just realized i don't need to debug it using my whole application, I tested the line of code and "result" ends with a NaN, does someone knows why?

Edited by lomateron

Share this post


Link to post
Share on other sites


I tested the line of code and "result" ends with a NaN, does someone knows why?

 

Generally that would mean that b is zero.

 


result = float(frac(a/b)==0.0f && c)

 

An equality comparison against a special floating point value is always a red flag. Due to limited floating point precision, the result may not be exactly equal to 0.0f

Share this post


Link to post
Share on other sites

now I conclude it was just paranoia, the debug thing shows that the returned value is a NaN when rendering to a UNORM8, when rendering to a FLOAT32 it doesn't shows any value,the debug jumps all the code till the end. but I can see the color and new value of the pixel, and the results I expect are correct.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!