I make a new topic for this one to avoid clutter. In the old one I asked for the brightness of specular reflections. Solved that problem using a micro-facet type BRDF as mentioned in a SIGGRAPH course paper from 2010 found on the Internet. The results are acceptable and the specular highlight works well. The problem I'm facing now is the excessive Fresnel based reflection from the environment map. Currently I use a code like this:

vec3 envMapDir = vec3( reflect( fragmentDirection, normal ) );
vec3 reflectedColor = vec3( textureLod( texEnvMap, pMatrixEnvMap * envMapDir, envMapLodLevel ) );
float reflectDot = max( dot( -fragmentDirection, normal ), 0.0 );
vec3 reflectivity = mix( vReflectivity, vec3( 1.0 ), vec3( pow( 1.0 - reflectDot, 5.0 ) ) );
outColor.rgb = mix( outColor.rgb, reflectedColor, reflectivity.rgb );

Hence in general it's Schlick's Fresnel as outlined in the SIGGRAPH paper using view and normal vector as parameter.

This leads though to the problem that anything receives a high rim-reflection no matter if it is a shiny piece of metal or a dull piece of weathered concrete. What's missing is the inclusion of the roughness of the surface as it is used in the specular reflection code. But what is the physically correct/reasonable way to use the roughness for environment reflection? At 0 the surface is fully sharp and should apply the full Fresnel effect. For 1 the surface is fully rough and there should be no Fresnel effect since the probability that a micro-facet looks into the right direction is 0. A linear blend doesn't seem to work (I use the formula ap=2.0/(roughness*roughness+0.00002)-1.5 so roughness is more or less linear). The SIGGRAPH paper unfortunately doesn't state anything concerning this problem.

If I understand the problem, then to correctly solve this, you've got to take your missing parameters into account when building the env map itself. Each single texel in the env-map corresponds to a cone of incoming light -- some of the rays in that cone will 'have more fresnel' than others, so you'd have to calculate the fresnel factors for each of these rays individually when you are building the cone (i.e. when writing the "blurred" texel into your env-map). This would also require one env-map per normal though, so it's obviously not realtime.
Seeing this isn't practical in games, there's no doubt another "close enough" answer.

Shouldn't this be possible without touching the environment map? The env-map I render dynamically depending on certain rules. But the problem affects only the surface itself. Hence if the surface is smooth like metal or rough like concrete the strength of the Fresnel reflection changes as does the shape and intensity of the specular reflection. What I don't get is how to apply the same effect to the Fresnel reflection that affects already the specular highlights. Both use a Fresnel computation just with different vectors. For the specular highlight the Fschlick uses normal and half-view vector. For the environment reflection Fschlick uses normal and view vector. So far both seem similar but this is only the Fresnel effect. In the specular highlight the roughness comes into play with the normalization factor which tones down or brightens up the highlight. For the environment map case I'm missing this parameter. From my understanding there should be an influence too which is not yet influenced by what is reflected (the content of the environment map) but how strong the reflection is. Roughness I use already to blur the env-map choosing a mip-map level but the missing problem is this toning down of the reflection.