I'm not really sure why this is (and what do you mean by "or else you're back at a Lambertian model"). It seems to me that a Lambertian, ideally-diffuse surface implicitly assumes that a) the apparent luminance is the same from all viewing angles (that is, light leaves the surface according to the law of cosine with the viewing angle) and b) that this luminance is determined according to the law of cosine with the light angle. There's not really anything explicitly preventing me from violating a) without taking into account the light angle.
Perhaps I can answer this one. If you don't make the distribution of light leaving the surface uniform, how do you determine what directions get more light than others?
I guess you could send more light along the surface normal direction and less in more tangential directions, or the other way around: But this would probably just look weird and unphysical.