As mentioned in my previous post, here is the way I've settled on handling weighting and blending the cone tracing results. For anyone following along, you'll notice that I decided not to bother with how much the cone sphere intersected the hi-z buffer. All the various combinations I tried with that approach gave bad results with a lot of artifacts. If someone else gets the weighting working properly and wants to share, I'd like to revisit it, but for the time being I'm content with what I have below. You'll notice I've brought in another new parameter to the function. The gloss is simply what you'd store in a gloss map (for me it's actually [1.0f - roughness], but you get the point). I simply use this as a further weighting on the output color. I noticed previously that even when I had a roughness of 1 (very rough), I was still getting more reflection than I would have expected, even at non-grazing angles. This new parameter helps clean that up nicely and fades more smoothly into your backup environment maps the rougher the surface becomes. I have it commented in the code, but it's worth noting that I leave gloss out of the alpha calculation. The more terms you add to that value, the further down you drive it for a particular iteration, and I've found that I get better results by just using the visibility and attenuation values. If you're implementing this effect, I encourage you to sub values in and out and see which works best for your needs.
One of the issues I would like to address in the future is the color buffer convolution stage. While the fading and blending helps this to an extent, you can easily get colors bleeding to areas where they shouldn't be due to the plain Gaussian blur used in creating this buffer's resources. In my setup, I also need to more properly blend this effect with the local cubemaps (most likely what I will tackle next).
As would be expected, this effect tends to work better the more depth information you have. For example, I've been posting images of the effect running in an outdoor scene with a few blocks and other programmer art, but largely an empty space. There are a lot of ray misses and some large depth discrepancies, and that results in some hard edge artifacts even with rougher surfaces. In the sponza scene with more "going on", I've found this technique to perform better aesthetically. I will post images of the same scene I've been using below, and later on after some more testing and polish I will try to get some out showing it running in sponza.
A great suggestion that Bruzer100 gave me was to not let the effect run to far. In other words even in the ray tracing steps you want to stop and return a non-hit after the ray has traveled a certain distance. I've made this into a constant buffer variable so I can update it per scene, and is the approach I would suggest be taken. This helps clean up a good amount of artifacts, and lets your fallback environment maps take over completely at known distances.
float4 coneSampleWeightedColor(float2 samplePos, float mipChannel, float3 rayStartVS, float radiusSS, float gloss)
{
// sample center and take additional sample points around the cone perimeter and inside to better blend the color result
float3 sampleColor = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos, mipChannel).rgb;
float3 sr = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos + float2(radiusSS, 0.0f), mipChannel).rgb;
float3 sl = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos - float2(radiusSS, 0.0f), mipChannel).rgb;
float3 sb = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos + float2(0.0f, radiusSS), mipChannel).rgb;
float3 st = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos - float2(0.0f, radiusSS), mipChannel).rgb;
float halfRadiusSS = radiusSS * 0.5f;
float3 srh = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos + float2(halfRadiusSS, halfRadiusSS), mipChannel).rgb;
float3 slh = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos - float2(halfRadiusSS, halfRadiusSS), mipChannel).rgb;
float3 sbh = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos + float2(halfRadiusSS, halfRadiusSS), mipChannel).rgb;
float3 sth = colorBuffer.SampleLevel(sampTrilinearClamp, samplePos - float2(halfRadiusSS, halfRadiusSS), mipChannel).rgb;
float3 blendedColor = (sr + sl + sb + st + srh + slh + sbh + sth + sampleColor) / 9.0f;
float visibility = visibilityBuffer.SampleLevel(sampTrilinearClamp, samplePos, mipChannel).r;
float depth = hiZBuffer.SampleLevel(sampPointClamp, samplePos, 0.0f).r;
float3 rayEndVS = viewSpacePositionFromDepth(samplePos, depth);
float distanceTraveled = length(rayEndVS - rayStartVS);
// distance squared caused the effect to fade too fast for larger objects in the distance
float attenuation = distanceTraveled == 0.0f ? 1.0f : saturate(1.0f / distanceTraveled);
// try mix and matching what goes into the alpha component - don't want to drive it down too much
return float4(blendedColor * visibility * attenuation * gloss, visibility * attenuation); // gloss intentionally left out of alpha
}
New screenshots:
https://www.dropbox.com/s/pze4pi5k9kb8d5f/screenshot_20.png?dl=0
https://www.dropbox.com/s/dfxravl5ghbkhia/screenshot_21.png?dl=0
https://www.dropbox.com/s/bbkiuzsycj6d8l3/screenshot_22.png?dl=0
https://www.dropbox.com/s/kciy1pxvfap1pgn/screenshot_23.png?dl=0
https://www.dropbox.com/s/2xg9amomoj25kyn/screenshot_24.png?dl=0
https://www.dropbox.com/s/2n1tisaqlt028k5/screenshot_25.png?dl=0
https://www.dropbox.com/s/ty4xdomw2ppkzbl/screenshot_26.png?dl=0
