if you apply a rotation matrix to the camera matrix it will always rotate around the world origin. so it should be enough to translate the camera along the negative position of the object, rotate, and then translate back.
another way would be storing two angles (relative to the object), set cam to identity, rotate around them, translate backwards (in cams local space) and translate along objects position (in world space).
or of course your cam matrix is always relativ to the object. in that case, set it to identity when its attachted to the object, translate it back as much as it should be away from the object and apply all rotations as usual. just dont forget to create the final matrix by multiplying the cams matrix with the objects matrix. that way all transformations to the object will automatically be applied to the cam.