What hex value will round up? For instance should 0x18 round up to 0x2 or down to 0x1?

I'm writing my own vsprintf equivalent function (don't ask), and I'm implementing the last type specifiers %a and %A where when the precision is less than the length of the mantissa the mantissa needs to be rounded off.

In base 10 half of 10 is 5 and 5 rounds up so in base 16 (hexadecimal) 8 is half of 16 and logically I'd have thought that meant 0x18 would round up to 0x2, and that's the way I implemented the rounding, if digit > 7 (ie 8 to f) then round up.

But when comparing to vsprintf there's an inconsistency, using the float -0.00037944390f and passing it to vsprintf (sprintf will probably do the same) with %a it will come out as "-0x1.5b98a2p-12", as it did with my implementation. But when the precision is 3 ("%.3a") vsprintf doesn't round up the 8 and outputs "-0x1.5b9p-12" but mine does round it up and instead outputs "-0x1.5bap-12".

It seems that the MS implementation is rounding hex values from 9 and up which doesn't make much sense, could someone explain why it doesn't round from 8? Or (as highly unlikely as it is) could it be that the MS implementation is wrong?

I had thought of posting all this originally but I figured since it was basically me vs microsoft I might get biased answers saying that whatever MS does is right without any actual knowledge of the actual answer ;)

Edit: Almost forgot a very important bit of info: I'm using VS2008 and it's implementation of vsprintf.

Whoops copied mismatching values >_<. The float value for it should have been -0.00033149359f, I'd accidentally run till the 2nd discrepancy when grabbing that float value but gave the strings for the fist one. I'm just using ~100000 random numbers and verifying that my function and vsprintf output the same strings when given the same values.

Both output "0x1.8de032p-12" as you said for -0.00037944390f. But nevermind, that also still has an 8 in it so you still get the same discrepancy but at 0 precision ("%.a"), vsprintf outputs "0x1p-12", mine outputs "0x2p-12".

Quote:Original post by frobMoving on, the full expansion of the value is 0x1.8de032p-12 (as above). So if you specify a precision of 3, you should expect three values on the right of the decimal point. This means you would round the fourth value, which in this case is '0', and if it were eight or larger you would increment the third and final value for display.

That's exactly what I thought should happen and how mine is implemented, BUT vsprintf will only increment the third value if the fourth is NINE or more, not eight.

Quote:Original post by ibebrett
technically when you round, you should round 0.5 down, because 4.99999(forever) is actually 5, which rounds down. the chances of this actually making a statistical difference are stupidly low

Actually 0.5 is perfectly handled by floats as 0x1.0p-1, but I know what you are getting at sometimes a the faction can't be fully resolved and wouldn't round correctly.

A friend of mine tried it in GCC and apparently that does the rounding at 8 like my code does. Looks like it could be a bug with the MS implementation of vsprintf :/