I should include that I also tried casting to uint and back as well, but that had no difference. The negative flip isn't held where I'm translating to int and back to byte.
I should also include that speed is not important here. This is for an editing tool to put the textures together. if it takes 20 minutes for 525 permutations, I'm ok with that. I just need it to work.