Jump to content

  • Log In with Google      Sign In   
  • Create Account


#Actual!Null

Posted 09 October 2012 - 02:19 PM


Actually, you're running out of precision in that operation. -255 * 409 is -104,295, but int16_t only has 16 bits of storage, which means it can only store values in the range −32,768 to 32,767. So what happens? Your intermediate result gets chomped down to 16 bits *before* the divide happens, and the end result is a weird value due to overflow.

What should you do about it? Use a bigger data type.


Could you please elaborate on that? What exactly is the factor that causes the calculation to be processed in 16-bit? Shouldn't the program detect that its running out of precision and automatically use a bigger (i.e. 32-bit) temporary buffer?

Which variables exactly should be increase in size?
EDIT: Is there a way to increase the temporary buffer size without changing the variable data types?


That's not how it works. Yes there is something that causes the calculation to be processed in 16-bit...its the fact you told it that your values are all 16 bit signed integers. when you get an overflow, instead of crashing your program. The value gets wrapped around. so when you reach the max value, it goes back to 0 and starts counting up again.

Is there a reason you chose int16_t or did you copy it from somewhere. Like I said change it to int32_t or higher and see the results

#1!Null

Posted 09 October 2012 - 02:18 PM


Actually, you're running out of precision in that operation. -255 * 409 is -104,295, but int16_t only has 16 bits of storage, which means it can only store values in the range −32,768 to 32,767. So what happens? Your intermediate result gets chomped down to 16 bits *before* the divide happens, and the end result is a weird value due to overflow.

What should you do about it? Use a bigger data type.


Could you please elaborate on that? What exactly is the factor that causes the calculation to be processed in 16-bit? Shouldn't the program detect that its running out of precision and automatically use a bigger (i.e. 32-bit) temporary buffer?

Which variables exactly should be increase in size?
EDIT: Is there a way to increase the temporary buffer size without changing the variable data types?


That's not how it works. Yes there is something that causes the calculation to be processed in 16-bit...its the fact you told it that your values are all 16 bit signed integers. when you get an overflow, instead of crashing your program. The value gets wrapped around. so when you reach the max value, it goes back to 0 and starts counting up again.

Is there a reason you chose int16_t or did you copy it from somewhere. Like I said change it to int64_t or higher and see the results

PARTNERS