You might be able to tweak Visual Studio's options to make it less bloated/slow, as well. Perhaps one particular setting is causing your pain?
This is indeed true, when I first researched the slowdowns (2005 / 2008 or so?) it seemed others had had the same problem, and I seem to remember it looked like it was caused by the background parsing of the source files, and perhaps some dodgy multithreading. I seem to remember I could improve it but the problem was still there. And the performance has been bad in every version I've tried since '.NET(!)' .. 2003, 2005, 2008, 2010 and 2013.
It might partly be to do with how accurately the parser is trying to map the source files, with users complaining that it has to be 'perfect'.... Early IDEs I suspect just did a wild stab at the classes etc, which is fast, and mostly correct 90% of the time. But with the result of the compilation depending on the particular #defines when you compile a file in c++, if you try and be accurate, you could be there forever as the effects of a small change propagate through the codebase. But still, other IDEs manage it.