So rather than using constantly 32x32 tiles for the middle part of the autotile, I would like to use other patterns like 2x2, 3x3, etc.
But I know how complex it is, so I try to ask my questions at the most basic level as possible.
And since your keys are 0-255 with no gaps, why do you need a hash table? A 1-d array would serve you just as well...
It's because the keys are the bit signature for all tile possibilities, so the same tile id can be used for many type of connections depending of the adjacent tiles around the middle tile.