After loading the Sponza Scene, with ALL textures and all meshes, the system hangs for 2-4 seconds (Can't move anything, could be the GPU hanging as it is processing all the stuff, maybe caching?), what could be the cause of this, and this is only when the first frame starts.

Set breakpoints and step through your code to see where it hangs. If there is no single point where it hangs for more than 1 seconds, its the amount of textures and meshes that need to be processed - if you e.g. load the meshes in a human readable format and not binary, it can take quite some time. Also try hitting F12 while running in the debugger and while it hangs, this will jump you to the code-point where the application currently is at - if nothig happens (don't know if this is works only for visual studio though), there is some external hang like on the GPU.