I'm not a photographer but I guess I actually mean tracking rather than panning, i.e the camera translates rather than rotates. I'll call it that from now on.

If you wanted to do it the other way, keeping the camera on the same line from the origin, you would have to rotate the camera to re-centre the box. I don't think there is a direct solution for the distance, but you could try an iterative method.

Do the calculation as before, but instead of translating, rotate the camera to look at the box centre plus the tracking vector. Then recalculate the distance using the new camera rotation. Move to that distance. Repeat a few times. I think the error will get smaller each time.

I'd recommend getting it working with camera tracking first.

I was hoping to avoid that but it looks like that's what I have to do.

So for future reference, should anyone have the same question, what I did to solve this was, in a loop, center the box and take a step closer to the box. If the box is too big (being cutt off) take a step back so the entire box is on screen and make your steps smaller. Keep doing this until the box is big enough (99% or 99.9% of the screen or whatever you want; you can even make it 50% if you don't want to maximze it).