I think I know why it's happening - my maximum ray depth is 5, and the rays probably get stuck in the cavities and never collect any colour
you are right, that sounds like the reason.
The way to fix it is to not limit your rays to a particular depth, but to reduce the amount of contribution on every recursion.
every surface in the real world ( and I mean also mirrors, glass, air, chrome...) have some loss in transmission of light. this can be simulated the easiest if you reduce the contribution they do by some amount per recursion level. every mirror ball would reduce it to e.g. 98% (you could set this value per material), it would take a while, but with increasing recursion it would end up by e.g. 1%
and once a ray reaches 1%, you could assume it won't contribute anything. (I think you can calculate the maximum recursion by taking the log of the canceling value and divide by the log of the reduction value e.g. log(0.01)/log(0.98) to estimate the worst case and maybe tweak both values to restrict the time consumption, instead of looking up realistic values)
that way you won't trace rays that have barely any contribution and you will trace important rays instead of canceling them due to some recursion limit. and the reduction of contribution might lead to a more realistic look.