I think I know why it's happening - my maximum ray depth is 5, and the rays probably get stuck in the cavities and never collect any colour
you are right, that sounds like the reason.
The way to fix it is to not limit your rays to a particular depth, but to reduce the amount of contribution on every recursion.
every surface in the real world ( and I mean also mirrors, glass, air, chrome...) have some loss in transmission of light. this can be simulated the easiest if you reduce the contribution they do by some amount per recursion level. every mirror ball would reduce it to e.g. 98% (you could set this value per material), it would take a while, but with increasing recursion it would end up by e.g. 1%
and once a ray reaches 1%, you could assume it won't contribute anything. (I think you can calculate the maximum recursion by taking the log of the canceling value and divide by the log of the reduction value e.g. log(0.01)/log(0.98) to estimate the worst case and maybe tweak both values to restrict the time consumption, instead of looking up realistic values)
that way you won't trace rays that have barely any contribution and you will trace important rays instead of canceling them due to some recursion limit. and the reduction of contribution might lead to a more realistic look.
The keyword for this is "russian roulette" (google it!) and is indeed a way to go as deep as necessary (in the sense that the final image will be unbiased, exactly as if you had used infinite depth for all your rays). Usually though it is only turned on after two bounces because it tends to increase variance (being a form of rejection sampling).
I should mention that it is only unbiased if it is properly implemented, and there are a bunch of variants floating around the internet (as usual when it comes to things like path tracing) which may or may not be equivalent so tread carefully.