The reason is when you alt-tab, you lose exclusive access to the graphics hardware and the device needs to be reset before you can get it back. So all dynamic resources have to be released, the device reset, and the resources re-acquired.
This is not an issue when your game is running in Windowed mode as it didn't have exclusive access in the first place.