Sounds like you're doing FPS/3rd person camera, in this case you need to offset both camera.CameraPosition and camera.TargetVector by same amount.

P.S. if player is holding RIGHT and DOWN he'll move (0.1f, 0.0f, 0.1f) which is longer distance than just (0.1f, 0.0f, 0.0f).

Sounds like you're doing FPS/3rd person camera, in this case you need to offset both camera.CameraPosition and camera.TargetVector by same amount.