The only thing I can think of that comes close to what you describe is multicolored noise on various objects due to interference of light at the surface, like oil film reflections but on a much smaller scale.
Interesting. By looking around my house, I actually discovered this when observing bright specular highlights on oiled woodwork, but something comparable also seemed to occur when I looked at the bright highlights produced by elaborate glass vases which were constructed like mosaics with many discrete glass sub-faces glued together (lot of caustic effects). I'm still trying to reproduce images. I just need to find the right time of day to take pictures, but I'm usually busy/distracted with programming.
At first I was thinking that you were simply observing dispersion, which makes sense for the glass vases, but that doesn't really happen on glossy wood. So it looks like a case of thin film interference, which is compatible with both situations (in the first case the film is the oil, in the second case it's the glue - or maybe it's actually dispersion in the second case and they just happen to look similar). I really cannot think of anything else at the moment. In either case, yes, what happens is only certain wavelengths of light make it to your eye depending on how you are looking at the object (others are either deflected away from your line of sight or eliminated by destructive interference) so if you move slightly the highlight will seem to change in color. This effect is of course independent of the lens system used to observe it. Is this what you see?