There will always be some form of distortion with a perspective plane projection, no matter what field of view you use. Higher fields of view will lead to extreme distortion at edges, and smaller fields of view will lead to loss of depth. Are you sure you are using the
factor correctly? In my raytracer, at a field of view of 45 degrees the distortion is not very noticeable except if I had tiny spheres at the edges, and quite acceptable, of course it depends on your standards.
Other solutions involve projecting the scene on a section of a sphere, which eliminate distortion but introduce other artifacts such as straight lines being projected curved and aren't as easy to compute... there is no perfect projection as you necessarily throw away information from a three-dimensional world to a two-dimensional plane.