And that dot product lighting in video games bypasses the concept of radiance and defines lights in terms of irradiance

What? No, the solid angle is always taken into account in direct light sampling (which is what games usually do). See that division by the distance to the point light squared? That's the solid angle part. The cosine term (dot product) does happen to cancel out for most BRDF's, though. Radiance and irradiance are two completely different concepts, irradiance does not take direction into account, it's simply the radiant energy flux density (i.e. per unit area) emitted by a surface (in every direction).

Your calculation using irradiance is technically correct for diffuse surfaces, but the only reason this is true is because such surfaces have constant reflectance (for diffuse surfaces, irradiance is equal to reflectance, this is what light baking actually does but it only works for diffuse-only surfaces, or generally surfaces where the reflectance happens to be independent of view angle). It does not work for more complex reflectance functions (BRDF's), this is what simple pixel shaders do:

set the lighting term to zero
calculate the surface normal N

for every point light in the world:
-- calculate the distance D from the light to this pixel
-- calculate the view vector V (camera to pixel) and the light vector L (pixel to light source)
-- evaluate the BRDF for (L, V) as reflectance ratio R (both for diffuse and specular)
-- divide everything by D^2 (inverse square law) and multiply/divide by dot(N, L), PI, as needed
-- add this to your "lighting term"

add ambient, light-independent stuff
modify your pixel's color depending on the lighting term calculated
 

Computer graphics shaders aren't a role model of clarity, generally they are not concerned with making sense, they just need to be fast and correct. Sometimes you'll see things like the inverse square division missing as it was accounted for elsewhere, the PI division was baked outside the shader, and so on.. shaders probably aren't the best way to learn radiometry.

I think it's really the notion of BRDF that you are missing here, diffuse surfaces aren't everything and tend to simplify a lot of stuff which is actually relevant for every other type of surface. As for the BRDF, the concept is fairly simple, if a bit abstract - for a light vector L and a view vector V, the BRDF returns the fraction of light incident to the surface along vector L (note this is radiance) which will be reflected along vector V.

PS: in the pseudocode above you also need to check if the point light is occluded by geometry, if so you need to cast a shadow instead. And you will notice this is not physically accurate, as it is ignoring the light contribution from light bouncing around the world and only considering direct lighting contribution from the point light sources themselves.

Area lights are in theory handled like an infinite number of point lights clumped together (but in computer graphics different methods are used, obviously).

Technically N•L is outside of the BRDF, and is a part of the rendering equation which is multiplied by the BRDF.

I think my problem stems from trying to define radiance at a point on the surface
which is impossible ... you need the view direction also

Yes, "radiance at a point" doesn't make much sense. Radiance is precisely defined as radiant exitance per solid angle (for computer graphics, read: "in a given direction"). You can sort of think of radiance as the flux through the cross-section of a cone originating from a surface in some direction. Note the flux decreases with cross-section area, and the cross-section area increases with distance squared (inverse square law). Whereas irradiance is just the total amount of light which falls onto a surface (per unit area).

This isn't a very rigorous way of thinking about it, but if it helps...

No, stop trying to fit irradiance into all this! For diffuse surfaces, the BRDF is equal to the constant (up to wavelength) albedo:

BRDF(V, L) = albedo;

And the rendering equation tells you that:

Radiance = (emitted light) + integral of directions L over the hemisphere centered of the surface normal of BRDF(V, L) * L * dot(N, L) dL

The integral itself includes the solid angle term, but since unless you are doing global illumination you are not evaluating the entire integral but only sampling a few point lights, you need to add it manually. This is basically what the pseudocode I gave above does, it evaluates the rendering equation's integral at a few discrete point lights (so the integral becomes a sum, as expected).

As you can see irradiance just doesn't come up when trying to perform ordinary lighting. It may come up when doing irradiance caching or baking light maps but if you are just doing simple lighting calculations, it is not something you need to think about.

You have the right idea but are just confused, I think. Do you understand the rendering equation? If not, try reading these links which I found useful myself: http://www.rorydriscoll.com/2008/08/24/lighting-the-rendering-equation/

BRDF is defined as a function which returns the ratio of incident _irradiance_ to outgoing radiance ...

Cirradiance = Clight * dot(N, L) / pow(distance, 2); // Clight is radiance 

my computer graphics book says:

E = R * dot(N, L) * differential_solid_angle 

which is the same thing ... so I really don't think I am wrong about this part?

That is the irradiance incident on the surface from the point light source's direction only, correct. And then you multiply that by the BRDF to work out how much of this irradiance (intensity: watts / surface area) is actually reflected towards any given direction (= exitant radiance) - probably the camera's view angle.

To be perfectly correct, Clight is not the radiance. It's the light source's power. Then the radiance towards the surface you are trying to work out the irradiance of is:

Clight / pow(distance, 2) // this is the radiance - no need for the area since this is a point light source

And you multiply by the cosine term to obtain the irradiance from this point light source (by taking into account the projected of the irradiated surface)

Now the book's version is slightly more general. With a differential solid angle you can do stuff like integration, whereas with your version you are restricted to dealing with point light sources as the distance requires knowledge of the surrounding geometry. This is an important distinction.