I still don't get how to correctly combine the reflection color sampled from the cubemap with the diffuse and specular color of my material.
If I just add the reflection to the (diffuse + specular) what I get is that the object is completely reflective and you won't even see any diffuse color at all.
On the other hand if I multiply the reflection color with the specular term and e.g. have a material that is highly glossy, this reflection will hardly be seen at all when the highlight is so small.
So as you explained multiplying by the specular term doesn't work, as expected, because the reflection color has nothing to do with the specular term of a light.
Next lets think about the diffuse term:
Multiplying by the diffuse term doesn't make sense either because what happens if you have a glossy black material?
When you multiply the reflection color with the diffuse color (0,0,0) you will loose the reflection which isn't correct right?
That leaves you with the option to simply add.
If the reflection color is too strong maybe you should be using a lower specular color in the Fresnel calculations...
Also what about shadows and occluded parts ?
That's problem I haven't looked into yet, and I guess to have an accurate result some form of raycasting will be needed.
I believe you have can use ambient occlusion (eg SSAO) to approximate occluded pixels.
Maybe I'm just thinking wrong but what I'm trying to achieve is something like this:
or the reflections on the ground here: http://seblagarde.files.wordpress.com/2011/12/locallightingcube.png
Those spheres are metallic so the specular color isn't monochromatic, which changes the color of the reflections since you use the specular color in the Fresnel calculation.
Also I understand that reflections from the env map are basically "specular" so in my engine would I use this reflection color on every object ? And if I wanted an object that is completely reflective from all sides would I consider this to be a separate material type ?
Since the environment map color doesn't depend on any lighting it makes sense that it calculations should be done in a separate shader from the lights.
If you have multiply lights in the scene, make sure that you're not adding the environment map color multiple times to each pixel.
For example, if you have a deferred renderer, since you run the lighting shader once for each light, you need a separate shader to add the environment map color just once.
Check this article, for some info on how to make glossy reflections.