I was able to fix it!
In my intersection function for the "world" I had an output parameter for the distance to the closest intersection, but I never saved that distance in the output parameter. This fixed the weird pattern you can see in the previous post.
I got horizontal black lines in the image as well, which were due to the fact that my intersection functions did not take rounding errors into account yet. So now I make use of the Ray's minimum t value to determine whether an intersection is considered valid.
Thanks for the help, everyone!