Yes, "radiance at a point" doesn't make much sense. Radiance is precisely defined as radiant exitance per solid angle (for computer graphics, read: "in a given direction"). You can sort of think of radiance as the flux through the cross-section of a cone originating from a surface in some direction. Note the flux decreases with cross-section area, and the cross-section area increases with distance squared (inverse square law). Whereas irradiance is just the total amount of light which falls onto a surface (per unit area).
I think my problem stems from trying to define radiance at a point on the surface
which is impossible ... you need the view direction also
This isn't a very rigorous way of thinking about it, but if it helps...