Just because you use C++ doesn't mean that you do have to do lower level work. There are loads of C++ libraries and engines out there that are higher level than XNA.
C# for me is a big timesaver and makes you more productive
XNA is far easier to get working on a game, without dealing with much of the lower-level work of initializing devices, managing vertex buffers, etc.
Pick the language you prefer and pick the library you feel most comfortable with. Perhaps only use DirectX or OpenGL (With either C# or C++) if you want to go into the technical stuff.