Okay, lets deal with the panning first.

So I've calculated a horizontal panning distance of 147 and a vertical panning distance of -448. So this means I need to pan along the camera's x-axis 147 units to the right and along the camera's y-axis 448 units down (first flag, it looks like I should be panning to the left, not the right). I calculate the normalized vectors that represent the camera's x and y-axis and multiply those vectors with the corresponding magnitudes and now I have the world coordinates.

Question, when you say pan do you mean move the camera? That's what I think when I hear pan. The problem with this is that the camera is positioned such that it is viewing the box at the given heading and elevation, if I move the camera this is now no longer the case. IE: The camera is something like 545 units up in the air as part of the 45° elevation, if I move the camera down the world version of 448 units, the camera is know, like, 15° elevated. The above picture is the camera LOOKING AT the calculated world coordinates of (147, -448).