Sounds like you hit your video cards memory limit and the drivers are now using system memory - which is also why your frame rate tanks. Task Manager only shows system memory usage, not the memory internal to the video card.
Good point. So it turns out that it was to do with my voxel visualizer that was causing the massive increase in system ram. I've turned that off and it doesn't seem to have any effect on framerate.
Looking at gpu ram, it makes sense now - 64 voxel depth (with all other resources) uses up about 750mb. This increases to 1.8gb when using 512 voxel depth.