Jump to content
  • Advertisement
Sign in to follow this  

Integer scaling

This topic is 2132 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi, this is probably really easy, but..


how do I best do a fast, accurate integer scaling (0.0 - 1.0) of 2 unsigned chars (8 bit values)?

[Note: this is the same problem as doing accurate alpha-blending. But it is also useful in other areas, e.g. audio.]




a = 100; // test number

b = 250; // scale factor 0 - 255, which should correspond to a 0.0 to 1.0 factor


result = (a * b) / 255;


However in this case the divide by a non-magic number (2, 4, 8, 16, etc) is presumably not as good as dividing by 256 which can be a bitshift? Although perhaps these days in most CPUs this is not an issue as memory bandwidth is more important?


And finally, what is the *best* way of doing this, allowing floating point stuff (and perhaps SIMD, maybe there is an instruction for it?).



Variations I've seen:


1) Just divide by 256 (the bitshift)


(and ignore the fact that 255 x 255 will come out as 254 as the result)


Pros: Works fast

Cons: Might be ok in many uses but isn't accurate.


2) result = (a * (b+1)) / 256;


Pros: Should be almost as fast as the above.

when b = 0

e.g. 255 * 1 = 255,

255 / 256 = 0 CORRECT

when b = 255

255 * 256 = 65280

65280 / 256 = 255 CORRECT


Cons: Are there some cases where this gives different results than the 'true' result? Not sure



Share this post

Link to post
Share on other sites

I can propose


3) result = (a * (1 + 2*b) + 256) / 512


None of those proposals give you the true result in all situations.


Option 1 disagrees with the true result in 47056 cases. 

Option 2 disagrees with the true result in 18760 cases.

Option 3 disagrees with the true result in 7720 cases.


What I am using as the true result is this:

result = (2 * a * b + 255) / 510


I like options 2 and 3 because they do the right thing for b=0 and b=255.


However, you have to ask yourself how expensive the division by 510 really is. Is your program noticeably faster with one formula than with another? If not, stick to the true formula.


[EDIT: Note that the compiler can use some tricks to compute the division by 510 as a multiplication by some magic constant and a couple of bit shifts. A detailed explanation of how this would work can be found here.]

Edited by Álvaro

Share this post

Link to post
Share on other sites

Wow thanks Alvaro. I'm sure a few of you have come across this exact issue.


I just was looking at it this morning, wondering at the best solution, as I'm doing an photoshop-like program at the moment so can sacrifice speed for getting the most accurate solution, but still want the 'fastest' accurate solution.


Must admit it makes my tiny brain go all fuzzy lol.


So is the


result = (2 * a * b + 255) / 510


more accurate than


result = (a * b) / 255



If so I'll change my code for this.

Share this post

Link to post
Share on other sites

The problem with (a*b)/255 is that it rounds down. If a is 89 and b is 209, a*b/255 is the integer 72, although the result of the division before truncation is actually 72.945098..., which is almost 73. I would say the correct value in that case is 73.


So the formula that I take to be `true' is



However, that formula involves converting between integers and floating-point numbers, which can be slow. But it's easy to fix if we move the +0.5 into the fraction like this:

int((a*b+127.5)/255.0) == int((2*a*b+255)/510.0) == (2*a*b+255)/510


Using (a*b)/255 results in an average darkening of the image by about half a level.

Share this post

Link to post
Share on other sites

(a*(b+1))/256 is wrong for numerous values, like a=185, b=186. Found by computer search.

Yes, and I posted above for how many values it is wrong: 18760 of the total 65536 possibilities. Of course I also found that with a little program.

Share this post

Link to post
Share on other sites

This is probably purely academic now, but it's kind of interesting lol smile.png


Alvaro, as your formula:


3) result = (a * (1 + 2*b) + 256) / 512


seems to essentially be 'diluting' the inaccuracy (as far as I can see), is it possible to use 'bigger numbers' to make the approximation method more accurate?


i.e. would using 1024 instead of 512 make it better? Essentially using whatever you could fit into say, a 32 bit unsigned int?


Haha my maths is clearly lost here lol biggrin.png

Share this post

Link to post
Share on other sites

The point of my formula (3) is that it is a proper scaling (i.e., of the form floor(a*fraction+0.5), where `fraction' is a real number between 0 and 1) and it does the right thing for b=0 and b=255.


But yes, you can get a more precise formula using larger numbers. This one seems to work perfectly:


Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!