#### Archived

This topic is now archived and is closed to further replies.

# Best fit in a paletted anti-aliaser

This topic is 6911 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hi, I''m writing a screen anti-aliaser for a 256 palette mode. Problem is, when I average the colours, and find the ACTUAL colour of what I need, the ''fitting'' process does not come up with the right colour. At the moment my best fit colour routine looks like this: byte fitcol(char *pal, byte r,byte g,byte b) { byte pr,pg,pb; int diff,sdiff=256,scol=0; for (int c=0;c<256;c++) { pr=pal[c*3]; pg=pal[1+c*3]; pb=pal[2+c*3]; diff=(abs(r-pr)+abs(g-pg)+abs(b-pb)); if (diff

##### Share on other sites
Try setting sdiff 256*3.
You see, the color black (0,0,0) and white (255,255,255) would have a difference of 765 (255 times 3), so you need a higher default value.

lntakitopi@aol.com | http://geocities.com/guanajam/

##### Share on other sites
Ah, sorry, I should''ve mentioned that my palette data uses only 0-63 in value for each of it''s RGB, so theoretically sdiff would be <192 at max.

Anyone else have a better algorithm that takes into account the closeness of Red, Green, Blue, specific to that colour?

Thanks and HELP!!!! once again,
FReY

##### Share on other sites
I can''t remember the algorithm right off the top of my head, but I know that a couple of years ago, I found a really good example of how to do this in the Allegro source. You could try looking there to see how they did it. The only details that I really remember at the moment is that the colors (red, green, blue) were weighted unevenly to account for the fact that the human eye is most perceptive of shades of certain colors over others.

Anyway, check out the Allegro source code and you should find a really good example of how to do it.

##### Share on other sites
Maybe we can hunt down the problem.
Is the color blatantly WRONG, (i.e. green instead of blue), or is it just not close enough?

lntakitopi@aol.com | http://geocities.com/guanajam/

##### Share on other sites
Well, lets just say I tried to antialias some trees, and they came out blue.

The problem is, that in my palette, the difference between the desired colour and multiple colours in my palette is the same.

I''ve used this algorithm before and it "seemed" to work then, but maybe it''s just a problem with my palette. It''s definitely nothing wrong with the anti-aliaser. It''s just the "best-fit" routine. . . Ugh!!

Greetings,
FReY

PS. I will take a look in the Allegro source though

##### Share on other sites
The problem is your algorithm is finding the closest brightness, not color.

You need to keep each color channel evaluated seperately.

You could try squaring or cubing the differences, and adding those up. That may give better results.

Or, you could try evaluating based only on the largest of the three differences.

These are just some suggestions.

##### Share on other sites
I don''t think that squaring/cubing them will work, though... you have to have an independant difference for R, G and B.

lntakitopi@aol.com | http://geocities.com/guanajam/

• 10
• 33
• 29
• 9
• 15