I'm simply wondering how to keep the contribution physically correct and variable depending on the environment.
I'm not sure it's realistic to make things like the sky brightness physically correct. Clouds might give off more light than a clear blue sky, but it's going to depend on so many factors. Probably more practical to just have some artistic control that is modulated by the overall sunlight.
For the "ground" part of your sky dome, the amount of light given off can be calculated assuming you know the approximate albedo of the ground and the amount of light coming from the sun or top half of the sky dome.
I did some experiments to try to improve my ambient lighting which may be of interest: