Jump to content

  • Log In with Google      Sign In   
  • Create Account


#ActualBismuth

Posted 09 October 2012 - 02:10 PM

Actually, you're running out of precision in that operation. -255 * 409 is -104,295, but int16_t only has 16 bits of storage, which means it can only store values in the range −32,768 to 32,767. So what happens? Your intermediate result gets chomped down to 16 bits *before* the divide happens, and the end result is a weird value due to overflow.

What should you do about it? Use a bigger data type.


Could you please elaborate on that? What exactly is the factor that causes the calculation to be processed in 16-bit? Shouldn't the program detect that its running out of precision and automatically use a bigger (i.e. 32-bit) temporary buffer?

Which variables exactly should be increase in size?
EDIT: Is there a way to increase the temporary buffer size without changing the variable data types?

#2Bismuth

Posted 09 October 2012 - 02:09 PM

Actually, you're running out of precision in that operation. -255 * 409 is -104,295, but int16_t only has 16 bits of storage, which means it can only store values in the range −32,768 to 32,767. So what happens? Your intermediate result gets chomped down to 16 bits *before* the divide happens, and the end result is a weird value due to overflow.

What should you do about it? Use a bigger data type.
EDIT: Is there a way to increase the temporary buffer size without changing the variable data types?


Could you please elaborate on that? What exactly is the factor that causes the calculation to be processed in 16-bit? Shouldn't the program detect that its running out of precision and automatically use a bigger (i.e. 32-bit) temporary buffer?

Which variables exactly should be increase in size?

#1Bismuth

Posted 09 October 2012 - 02:07 PM

Actually, you're running out of precision in that operation. -255 * 409 is -104,295, but int16_t only has 16 bits of storage, which means it can only store values in the range −32,768 to 32,767. So what happens? Your intermediate result gets chomped down to 16 bits *before* the divide happens, and the end result is a weird value due to overflow.

What should you do about it? Use a bigger data type.


Could you please elaborate on that? What exactly is the factor that causes the calculation to be processed in 16-bit? Shouldn't the program detect that its running out of precision and automatically use a bigger (i.e. 32-bit) temporary buffer?

Which variables exactly should be increase in size?

PARTNERS