Ray Tracing Shadows and Refraction

Hello, I'm making a ray-tracer. It will have reflection, refraction, and shadows. I started with spheres, because they are simple. Before I move to polygons I want these effects. There weren't really any good resources for making a ray-tracer in Python, (which is what I mostly use), so I just figured it out. As a consequence, I know precisely how every bit of it works, and am not just stitching together other peoples' code. Here's some screenshots of what I've got so far: http://geometrian.com/Gallery.php. It is pretty fast, even though it is in Python and not optimized. As you can see, I have succeeded at making a ray-tracer which renders shadows and reflections quite well. I am, however, noticing a problem implementing refraction. Refracting objects bend light. It has occurred to me that refracting objects should bend shadow rays too. As the picture demonstrates, a direct-line shadow ray (black) would hit a refracting object and be bent (bending black), not reaching the light. The point would be interpreted as being in shadow. An alternate indirect shadow ray (red) would actually illuminate it, so the conclusion that the point is in shadow is wrong. Because I want my ray-tracer to be as accurate as possible, it must support shadows and refraction at the same time. How can this be done? Thanks, Geometrian