in the case of multiple intervening walls, a "front to back" search - from pivot point back to camera - will be faster than a back to front search - from camera towards pivot point. front to back you're done as soon as you hit the first wall. back to front, you have to go all the way to the camera, noting the last wall hit as you go.
now what about when your back is against a wall?
So if you implement the ray-casting collision yourself you might be concerned about back-to-front/front-to-back etc, however most physics engines will use some sort of BVH or space partitioning to perform the ray check.
If your back is against the wall, then the algorithm I described will truncate the camera position to be very close to the "focus" point. In that case, you could override the camera positioning and zoom into a "first person" view.