I think when you play games, you should play games. AR games have narrow scopes, and are only usable in limited environment like laser tags or board games, and it only serves as an enhancement. You obviously don't want to be playing Frogger while crossing real streets. You will only have one life there, and there's no princess on the other side of the street.
Google glass itself is already a huge distraction to the person wearing it, and the people around him. It has limited practical purposes, but far usable as a video gaming console.
Oculus Rift is the only thing that I think has some potential as a gaming platform. It's something you wear while you play games, and play games only. You are not eating while wearing that. You are not taking a shower while wearing that.
I don't think you should not be doing anything else while playing games.