Jump to content
  • Advertisement
Sign in to follow this  
Skiller

Hexadecimal rounding

This topic is 3638 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Advertisement
I can't understand why you would want to do this. Can you give us more information?

The example you gave translates to 'should 24 round to 1 or 2?', where neither result seams particularly useful.

Share this post


Link to post
Share on other sites
I'm writing my own vsprintf equivalent function (don't ask), and I'm implementing the last type specifiers %a and %A where when the precision is less than the length of the mantissa the mantissa needs to be rounded off.

In base 10 half of 10 is 5 and 5 rounds up so in base 16 (hexadecimal) 8 is half of 16 and logically I'd have thought that meant 0x18 would round up to 0x2, and that's the way I implemented the rounding, if digit > 7 (ie 8 to f) then round up.

But when comparing to vsprintf there's an inconsistency, using the float -0.00037944390f and passing it to vsprintf (sprintf will probably do the same) with %a it will come out as "-0x1.5b98a2p-12", as it did with my implementation. But when the precision is 3 ("%.3a") vsprintf doesn't round up the 8 and outputs "-0x1.5b9p-12" but mine does round it up and instead outputs "-0x1.5bap-12".

It seems that the MS implementation is rounding hex values from 9 and up which doesn't make much sense, could someone explain why it doesn't round from 8? Or (as highly unlikely as it is) could it be that the MS implementation is wrong?

I had thought of posting all this originally but I figured since it was basically me vs microsoft I might get biased answers saying that whatever MS does is right without any actual knowledge of the actual answer ;)

Edit: Almost forgot a very important bit of info: I'm using VS2008 and it's implementation of vsprintf.

Share this post


Link to post
Share on other sites
Quote:
Original post by Skiller
But when comparing to vsprintf there's an inconsistency, using the float -0.00037944390f and passing it to vsprintf (sprintf will probably do the same) with %a it will come out as "-0x1.5b98a2p-12", as it did with my implementation. But when the precision is 3 ("%.3a") vsprintf doesn't round up the 8 and outputs "-0x1.5b9p-12" but mine does round it up and instead outputs "-0x1.5bap-12".

This at least makes a little bit of sense.

When I run this on Visual Studio 2008, I get the results:
printf("%a",0.00037944390f);
yeilds: 0x1.8de032p-12
printf("%.3a",0.00037944390f);
yeilds:0x1.8dep-12

Your numbers are significantly incorrect, the input should not yield the numbers you received.

Next, you have a double and not a float. Variadic parameter types are self-promoting, so you cannot have a float, char, short, or other promotable types.

Moving on, the full expansion of the value is 0x1.8de032p-12 (as above). So if you specify a precision of 3, you should expect three values on the right of the decimal point. This means you would round the fourth value, which in this case is '0', and if it were eight or larger you would increment the third and final value for display.

This means that the value 0x1.8dep-12 is correct, where the third value is not increased for rounding.

If instead you had an original value of 0x1.8de932p-12, then the nine causes the rounded result 0x1.8dfp-12, where the third value is increased for rounding.

Share this post


Link to post
Share on other sites
Whoops copied mismatching values >_<. The float value for it should have been -0.00033149359f, I'd accidentally run till the 2nd discrepancy when grabbing that float value but gave the strings for the fist one. I'm just using ~100000 random numbers and verifying that my function and vsprintf output the same strings when given the same values.

Both output "0x1.8de032p-12" as you said for -0.00037944390f. But nevermind, that also still has an 8 in it so you still get the same discrepancy but at 0 precision ("%.a"), vsprintf outputs "0x1p-12", mine outputs "0x2p-12".

Quote:
Original post by frobMoving on, the full expansion of the value is 0x1.8de032p-12 (as above). So if you specify a precision of 3, you should expect three values on the right of the decimal point. This means you would round the fourth value, which in this case is '0', and if it were eight or larger you would increment the third and final value for display.


That's exactly what I thought should happen and how mine is implemented, BUT vsprintf will only increment the third value if the fourth is NINE or more, not eight.

Share this post


Link to post
Share on other sites
technically when you round, you should round 0.5 down, because 4.99999(forever) is actually 5, which rounds down. the chances of this actually making a statistical difference are stupidly low

Share this post


Link to post
Share on other sites
Quote:
Original post by ibebrett
technically when you round, you should round 0.5 down, because 4.99999(forever) is actually 5, which rounds down. the chances of this actually making a statistical difference are stupidly low


Actually 0.5 is perfectly handled by floats as 0x1.0p-1, but I know what you are getting at sometimes a the faction can't be fully resolved and wouldn't round correctly.

Share this post


Link to post
Share on other sites
A friend of mine tried it in GCC and apparently that does the rounding at 8 like my code does. Looks like it could be a bug with the MS implementation of vsprintf :/

Share this post


Link to post
Share on other sites
Quote:
Original post by Skiller
A friend of mine tried it in GCC and apparently that does the rounding at 8 like my code does. Looks like it could be a bug with the MS implementation of vsprintf :/


No, it is unspecified.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!