Multiplying the reflection color with the ambient light (and ambient occlusion) does make it look good, but I guess now its intensity would depend entirely on the ambient light intensity in the scene.
I would simply multiply with the ambient occlusion factor (not the ambient light). The reflections might be to strong because the environment map luminance is much higher than the directional light intensity, and when you multiply the reflection color with the ambient light you scale it back and it starts to look better. Keep in mind that a physically based renderer usually requires some calibrating work to be done.
Also would I add the reflection to everything ? Or exclude it on materials like skin ? (not sure I can't remember having seen environmental reflections on human skin before in real life).
No need for that. Even though every material reflects light, there's no need to waste computation time calculating reflections for rough materials because the reflection won't be noticeable. For movies, there's time to add reflections on every material because you don't need to run at 30/60 fps, games, on the other hand, need to cut on details like that.
Skin is actually a good candidate for environment reflections, especially in bald people at grazing angles, but since its a rough material the reflections are so much "blurred" that it looks like a uniform color.
And what you said earlier about the cryengine screenshot of those sphere's that is a metallic reflection ? How is it different ? Do I just output specular and ignore diffuse completely ? You're saying the color of the spheres in that screenshot comes from the specular color ?
Yes, diffuse color can be ignored in metallic surfaces.
Example: Gold specular color (1.0f, 0.71f, 0.29f)
Even if HalfVector dot Light or View dot Normal = 1 the result of fresnel will be (1.0f, 0.71f, 0.29f).
Here's a screen of how it looks multiplied by the ambient and ambient occlusion:
The reflections are very faint now. Which makes perfect sense on the ground but maybe not on the highly glossy sphere on the right.
What specular color are you using in the sphere? You might need to increase it. Especially if the sphere is supposed to be metallic.
Also I'm not quite sure what those numbers are (0.2 being rough and 1.0 being glossy). In Phong and Blinn-Phong you have exponents ranging from 1 to infinity where higher numbers represent a smoother surface and in NDFs like Beckmann or GGX you have factors ranging from 0 to 1 where higher numbers represent a rougher surface. You are either using a pretty weird NDF / BRDF or you misunderstand something.
Its common to store a glossiness value in the range [0,1] in textures and then convert it to specular power using a function like this:
float specPower = pow(2.0f, 13*glossiness);