The issue with environment maps is, that you need more or less exact probes of the surrounding in dark areas, which is quite difficult in my game due to its dynamic nature. Env-maps simulate the reflection of the surrounding, similar to the specular highlights of close light sources. The problem occurs when the environment is dark, while the env-map displays a lit environment and acts as a light source (especially when using HDR), in this case the env-map immediately catch your eyes.
You can take a look at the screenshot. On the left side the environment map on the test sphere and the bands of the barrels is clearly visible. My new approach is displayed at the right side, where it looks similar when the environment is lit and is more or less turned off when the environment is dark.
I'm using a deferred render engine, therefor I blend the environment only in when it is close to a light source. Additionally I modify the factor by the angle between the light source and surface normal to dim the effect on unlit surfaces.
float normal_factor = dot(light_direction, surface_normal) * 0.5 + 0.5; float distance_factor = smoothstep(max_light_radius,min_light_radius, distance(light_position, pixel_position)); final_light += env_color.rgb * normal_factor * distance_factor;