It's not about memory it's about how much CPU time doing a lot of ray casting takes really.
May you elaborate more in detail and gives example pertaining to CPU time? Interesting stuff
As cr88192 pointed out it's really just the CPU having to do "more stuff." CPU's aren't magic, they work in linear time and the only limit to them really is how many things they can do over a period of time.
Often in games you have something like the logic processing stage of the game loop, this is where the CPU takes control, doing mathematical comparisons, collision detection, AI choices, ray casting in this case, so on. Basically it does everything to make game objects interact, subsequently the more time the CPU spends doing tasks the less time is available for other tasks. If you do 100k checks compared to 10k checks you are spending 10x as much CPU time on that.
CPU's these days are pretty robust, it takes a lot of math work to bog them down but in higher end games it certainly becomes a problem, if you think about something like an AAA fps you may have an entire level loaded with geometry, AI soldiers or something running around and interacting, and the game is constantly checking their collision with things near them as well as every other task.
This is why games are often considered "CPU" or "GPU" bound, because it is a linear step(one game may have very little for the video card to process but a ton of CPU work to process), for simplicity sake most games do the logic processing then the graphic updates. If the CPU is slow it may take a long time to do all the processing required, and then the GPU only has so much time to render to maintain the framerate. But that's what it comes down to, the two real ways to optimize when coding are to do something that takes -less- CPU time, or to do less things in general. As an example, one simple addition statement takes a lot less cycles than casting and checking 10k rays every logic update(which may be every 20 ms or something.)