I agree that people will pump up those around them to keep from having to be critical of themselves. But more often than not people will assume I'm really smart before I do anything one would describe as smart. Even before we have a conversation. I don't think it's something magic about me, or that I actually am just naturally a genius and everyone can sense it. I probably just act some way that reminds people of a cliche "smart person" as portrayed in the media. Or that's my only logical assumption. It could be something else I'm not considering.
As for being called a genius for programming and reading and enjoying math, unfortunately it's just a constant reminder that, at least in American society, most people are so inexperienced when it comes to actually honing a skill that when they see people with honed skills, it doesn't even occur to them that it was time, not a gift, that got the person to where they are.