Jump to content

  • Log In with Google      Sign In   
  • Create Account

help with result of line of code


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
4 replies to this topic

#1 lomateron   Members   -  Reputation: 342

Like
0Likes
Like

Posted 30 December 2013 - 12:43 AM

In HLSL 4.0 i have this line that is imposible for me to get the debugger to it(apart from having to wait around 15 minutes to build the solution in debug mode)

 

a=30030.0f;

b=11.0f;

c=true;

 

result = float(frac(a/b)==0.0f && c)

 

I expect  "result" to be 1.0f but I think "result" ends with value 0.0f, does anyone know why it ends with 0.0f, or is just paranoia? 



Sponsor:

#2 lomateron   Members   -  Reputation: 342

Like
0Likes
Like

Posted 30 December 2013 - 01:01 AM

I just realized i don't need to debug it using my whole application, I tested the line of code and "result" ends with a NaN, does someone knows why?


Edited by lomateron, 30 December 2013 - 01:01 AM.


#3 phil_t   Crossbones+   -  Reputation: 3915

Like
1Likes
Like

Posted 30 December 2013 - 01:37 AM


I tested the line of code and "result" ends with a NaN, does someone knows why?

 

Generally that would mean that b is zero.

 


result = float(frac(a/b)==0.0f && c)

 

An equality comparison against a special floating point value is always a red flag. Due to limited floating point precision, the result may not be exactly equal to 0.0f



#4 lomateron   Members   -  Reputation: 342

Like
0Likes
Like

Posted 30 December 2013 - 02:21 AM

Its NaN when :

a=30030.0f;

b=11.0f;

c=true;

Does someone know why?



#5 lomateron   Members   -  Reputation: 342

Like
0Likes
Like

Posted 30 December 2013 - 03:14 AM

now I conclude it was just paranoia, the debug thing shows that the returned value is a NaN when rendering to a UNORM8, when rendering to a FLOAT32 it doesn't shows any value,the debug jumps all the code till the end. but I can see the color and new value of the pixel, and the results I expect are correct.






Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS