You now have a 5-sided frustum with no near plane (it extends into infinity). Perform frustum culling and gather those objects into a list of objects that cast shadows into the player’s frustum.
This is the core of it all, and since I didn't spot any further elaboration on this in L. Spiro post, I will add up.
1- cull object from view of observer
2- cull objects from view of light in very screen space, Meaning, use AABB points of an object in screen space, but do not examine them only for being on screen (thus in light frustum) but also for being any of them over projected observer frustum in the light's projection space.
My concern is that if any of AABB points of an object is in screen space of light projection, it casts a shadow, but if none of its point is over projected observer frustum (in the light's projection), it does not cast shadow to observer frustum.
You would test if an AABB is in observer frustum by comparing only x,y components towards projected observer frustum x,y components. If any of them is positive, you still have a chance of it not casting shadow. But this chance is rather small, and I would render from there on, but you can still test it if you wish further, so you move to comparing the AABB point along with z component for being in or over the projected frustum (not behind it)- this is rather grewsome step and will not save you from many rander calls most of the time I believe.