To explain samoth's post a bit:
The view matrix is the inverse of the camera's world transformation. This is usually a composition of translation and rotation, perhaps a scaling. Assuming column vector matrices, this looks like so:
C := T * R * S
A decomposition of C into a translation T, rotation R, and scaling S, as shown above, is relatively easy. It is even easier if scaling is known to not appear, because then decomposition is just extraction of values.
Then the inverse is
V := C-1 = S-1 * R-1 * T-1
( T * R * S ) * ( S-1 * R-1 * T-1) = T * ( R * ( S * S-1 ) * R-1 ) * T-1 = I
Looking at the properties of the particular matrices, the following correspondences can be used:
( S(sx, sy, sz) )-1 = S(1/sx, 1/sy, 1/sz)
( R )-1 = Rt
( T(tx, ty, tz) )-1 = T(-tx, -ty, -tz)
So the inversion of such a matrix C can be replaced by a decomposition and matrix products of usual translation, rotation, and scaling matrices.