Take a long look at the actual vectors that make up your rotation matrix. Eventually you should notice (depending on how your game defines its axes) that the rotation matrix is nothing else but the forward, right and up direction vectors. No point in additionally storing angles. If you need to apply the rotation, just use the matrix, if you need to move, just use the vectors from the matrix. That's basically the benefit for using a representation that is more wasteful than others (angles, quaternions, two vectors).
One major downside if you keep accumulating rotations in a matrix is that small errors will also accumulate and force you to re-orthonormalize the matrix every once in a while.