you can perform an optimized 4x4 matrix inverse by transposing the 3x3 portion and doing the dot products to get the last row.
If you have a transformation that is a rotation and a translation, e.g. the 3d camera's position is t and it's rotation is R then it's world transform is,
x' = Rx + t, where R is a 3x3 matrix, x is a 3x1 matrix(column vector) and t is a 3x1 matrix(column vector). The output is x', your transformed vector.
To get a view matrix you want to bring all vertices in the frame of reference of the camera so the camera sits at the origin looking down one of the Cartesian axis. You simply need to invert the above equation and solve for x.
x' = Rx + t
x' - t = Rx
R-1(x' - t) = (R-1)Rx
RT(x' - t) = x // because a rotation matrix is orthogonal then its transpose equals its inverse
x = RTx' - RTt
So to invert a 4x4 camera transformation matrix, efficiently, you will want to transpose the upper 3x3 block of the matrix and for the translation part of the matrix you will want to negate and transform your translation vector by the transposed rotation matrix.