It looks correct. So I'm not sure what the problem might be...

Let's try one more thing:
Remove the normal mapping, so the normal of the ground is (0.0f, 1.0f, 0.0f) on all pixels, and just output the reflection color. So there're no changes in normals, diffuse colors and lighting.

The ground should be black near the camera and the reflection of the environment map should increase with distance around the camera.

P.S: I noticed that you're multiplying the reflection color with the shadow factor, which isn't correct because a pixel might be in shadow from the light but it still receives light from the environment. You might want to multiply it with the ambient occlusion factor though

Basically a material that is highly glossy (let's say = 1.0f) would have strong reflection intensity and a material that is very rough (= 0.2f) would then have very low reflection intensity. Does that make sense, or would it be "physically incorrect" somehow ?

A rough material reflects just as much light as a smooth one. The only difference is, that on the rough material the reflections are more scattered into all directions and not as bundled into a single direction. That gives the illusion of a highly smooth material being more reflective, but that's not true, it's just more or less "binary" as to how it reflects light (completely into one direction, nothing into another direction).

The only parameters needed to describe reflection are parameters to describe the materials microfacets orientations and positions (usually a single parameter called "Roughness" or "Glossiness") and the index of refraction of the material. The IOR is completely responsible for as to how much light is reflected from a microfacet which can be calculated using fresnel's equations. The easier and more common way to store this IOR is to calculate how much light would be reflected at microfacet normal incidence (aka specular intensity) and use this later on with Schlick's approximation of fresnel's equations which approximates them without the need for actual IOR values.

Each microfacet is a perfect mirror so light coming from one direction will be reflected to exactly one other direction. So how each microfacet is oriented doesn't affect how much light is reflected, only where it's being reflected to.

So overall it doesn't matter how rough your material is, the only thing that is actually responsible for how much is reflected, is the index of refraction.

You could actually derive a diffuse model from this microfacet model as well:
Simply multiply lambert with the integral over all the microfacet orientations of the percentage of microfacets being oriented into this direction multiplied with the percentage of microfacets actually visible from the light source multiplied with the complement of how much light is reflected from these microfacets (because the non-reflected rest of the light is getting scattered into the material and thus might get scattered out again into random directions as diffuse lighting if it's not absorbed).

Also I'm not quite sure what those numbers are (0.2 being rough and 1.0 being glossy). In Phong and Blinn-Phong you have exponents ranging from 1 to infinity where higher numbers represent a smoother surface and in NDFs like Beckmann or GGX you have factors ranging from 0 to 1 where higher numbers represent a rougher surface. You are either using a pretty weird NDF / BRDF or you misunderstand something.

Also would I add the reflection to everything ? Or exclude it on materials like skin ?

Everything has an index of refraction and thus everything must be reflective unless it has the same index of refraction as air (or whatever material the camera is in). Skin for example is pretty much as reflective as water is (the index of refraction doesn't differ that much), but is pretty rough in comparison to water.

metallic reflection ? How is it different ? Do I just output specular and ignore diffuse completely ? You're saying the color of the spheres in that screenshot comes from the specular color ?

Metals have index of refractions in the complex plane. If you calculate how much light is reflected using fresnel's equations, you get pretty high percentages. These are usually around 95% instead of at about 4% for dielectric materials. Also most metals unlike dielectric materials have highly varying index of refractions in the visible light spectrum. So green light might get reflected less than red light for example. Copper and gold are good examples of materials with varying index of refractions in the visible light spectrum. Also metals are highly absorbtive. The already low amount of light that is not reflected and gets scattered into the material gets absorbed so fast, that pretty much no light gets scattered out again as diffuse lighting. Keep in mind, that there is no difference between subsurface scattering and diffuse lighting.

Multiplying the reflection color with the ambient light (and ambient occlusion) does make it look good, but I guess now its intensity would depend entirely on the ambient light intensity in the scene.

I would simply multiply with the ambient occlusion factor (not the ambient light). The reflections might be to strong because the environment map luminance is much higher than the directional light intensity, and when you multiply the reflection color with the ambient light you scale it back and it starts to look better. Keep in mind that a physically based renderer usually requires some calibrating work to be done.

Also would I add the reflection to everything ? Or exclude it on materials like skin ? (not sure I can't remember having seen environmental reflections on human skin before in real life).

No need for that. Even though every material reflects light, there's no need to waste computation time calculating reflections for rough materials because the reflection won't be noticeable. For movies, there's time to add reflections on every material because you don't need to run at 30/60 fps, games, on the other hand, need to cut on details like that.
Skin is actually a good candidate for environment reflections, especially in bald people at grazing angles, but since its a rough material the reflections are so much "blurred" that it looks like a uniform color.

And what you said earlier about the cryengine screenshot of those sphere's that is a metallic reflection ? How is it different ? Do I just output specular and ignore diffuse completely ? You're saying the color of the spheres in that screenshot comes from the specular color ?

Yes, diffuse color can be ignored in metallic surfaces.
Example: Gold specular color (1.0f, 0.71f, 0.29f)
Even if HalfVector dot Light or View dot Normal = 1 the result of fresnel will be (1.0f, 0.71f, 0.29f).

Here's a screen of how it looks multiplied by the ambient and ambient occlusion:
http://d.pr/i/SuZh

The reflections are very faint now. Which makes perfect sense on the ground but maybe not on the highly glossy sphere on the right.

What specular color are you using in the sphere? You might need to increase it. Especially if the sphere is supposed to be metallic.

Also I'm not quite sure what those numbers are (0.2 being rough and 1.0 being glossy). In Phong and Blinn-Phong you have exponents ranging from 1 to infinity where higher numbers represent a smoother surface and in NDFs like Beckmann or GGX you have factors ranging from 0 to 1 where higher numbers represent a rougher surface. You are either using a pretty weird NDF / BRDF or you misunderstand something.

Its common to store a glossiness value in the range [0,1] in textures and then convert it to specular power using a function like this:

float specPower    = pow(2.0f, 13*glossiness);

No need for that. Even though every material reflects light, there's no need to waste computation time calculating reflections for rough materials because the reflection won't be noticeable.

So rough gold would be completely black? I don't know, roughness doesn't change anything about the reflectivity of the material, just the "blurriness" of the reflection. So on metals or at grazing angles you'll lose an incredible amount of lighting if you simply turn it off for rough materials. If anything I'd approximate rough reflection using an approximation via a diffuse term.

Its common to store a glossiness value in the range [0,1] in textures and then convert it to specular power using a function like this:
float specPower = pow(2.0f, 13*glossiness);

Oh, my bad. I thought he was talking about the specular power used in the BRDF. :D

So would it be correct to multiply this by the ambient occlusion ?

Ambient occlusion is low-quality shadowing information. Use it on anything non-directional, where you don't have better visibility information. So don't apply it to reflections or any point / spot / directional lights. Just to irradiance or other kinds of ambient terms, like diffuse sky illumination. Basically every source of light which comes from more than just a single direction (this includes area lights) could be multiplied with ambient occlusion. The more it spans the hemisphere over the surface, the more it's suited for ambient occlusion. Environment maps might span the whole hemisphere over the surface, but when used for reflections, only a single direction is used, so you shouldn't apply ambient occlusion to it. If the environment map is used for diffuse image based lighting, samples over the whole hemisphere are used and thus ambient occlusion should be applied. For rough reflections you might want to blend in ambient occlusion to some degree though.

And how about shadows ? Do I multiply or add them to the environment map color ? Or not at all ?

I'm not quite sure which shadows you're talking about. But basically you are "tracing" a reflection ray and the environment map helps you find the intersection. So there's no need for any kind of shadow map.

But even with a fresnel reflectance color of 0.028f (for skin) I'm seeing reflections even when not at glancing angle on the skin. Do you think the color of the env map is too bright ?

Either it's actually too strong or the reflection is not rough enough. Hard to tell without seeing the result.

Do you think the color of the env map is too bright ? I basically just took my skybox and prefiltered it with amd's cubemap gen. And I'm also already reading the env map as linear.

Maybe... You have to make sure that intensity of the env map matches the light sources in the scene...

So rough gold would be completely black? I don't know, roughness doesn't change anything about the reflectivity of the material, just the "blurriness" of the reflection. So on metals or at grazing angles you'll lose an incredible amount of lighting if you simply turn it off for rough materials. If anything I'd approximate rough reflection using an approximation via a diffuse term.

You're right, for metals the reflections shouldn't be disabled. I was talking about materials with both low specular colors and glossiness where the reflections aren't much noticeable, like the trunk of a tree, etc.

Ambient occlusion is low-quality shadowing information. Use it on anything non-directional, where you don't have better visibility information. So don't apply it to reflections or any point / spot / directional lights. Just to irradiance or other kinds of ambient terms, like diffuse sky illumination. Basically every source of light which comes from more than just a single direction (this includes area lights) could be multiplied with ambient occlusion. The more it spans the hemisphere over the surface, the more it's suited for ambient occlusion. Environment maps might span the whole hemisphere over the surface, but when used for reflections, only a single direction is used, so you shouldn't apply ambient occlusion to it. If the environment map is used for diffuse image based lighting, samples over the whole hemisphere are used and thus ambient occlusion should be applied. For rough reflections you might want to blend in ambient occlusion to some degree though.

Since the OP is using the same environment map for all objects multiplying the reflections with the ambient occlusion factor might help reduce artifacts like reflections of the environment on the bottom of the spheres near the ground where they should be occluded. This is just a suggestion though, since occlusion in environment mapping is a rather complex problem.