Are you sure your GPU has enough memory? System will silently transfer data between GRAM and RAM.
Your explanation reminds me of situation I had in a game I played: I had 60 FPS with static scene but whenever I turned camera it dropped to 10 FPS only because PC didn't have enough GRAM to fit all textures at once.
It should have enough, gtx650 has 1gb of dedicated ram and I have very few assets in gpu(~20-30 textures, most of them 256x256). But to confirm, is there a way to find out how much memory is left in GRAM?