• Create Account

## Multiply RGB by luminance or working in CIE ? Which is more correct ?

Old topic!

Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

18 replies to this topic

### #1MegaPixel  Members

241
Like
1Likes
Like

Posted 24 September 2012 - 03:40 AM

Hi guys,

I was studying through tonemapping operators and exposure control.

I've read several threads on gamedev about those topics and I've also read the MJP article on tonemapping on his blog and John Hable on filmic S curve.

Just one thing Is not clear to me: I understand that exposure control and tonemapping are two different things, but I wonder why someone is still multipling the RGB color by the adapted/tonemapped luminance L straight away.
Shouldn't be more correct to go in CIE color space Yxy, adapt Y to get Y' and then given Y'xy go back to RGB to get the new tonemapped color ?
Also, given that automatic exposure is a different thing with respect to tonemapping (which brings the Luminance values in [0,1] range in Reinhard) why in the Reinhard paper he basically calculates the geometric mean to get the average luminance in the context of a tonemapping process (I mean it's just part of the process but is still not the tonemapping step), that means that the relative luminance then can be used in the context of any tonemapping curve (not just reinhard) to have automatic exposure control ? And therefore shouldn't the relative luminance Lr calulation (the one before Lr / (1 + Lr), which is the actual tonemapped Lt ) consider the interpolated Lavg across frames in its calculation ?

Lr = (L(x,y) / Lavg)*a, where Lavg is the interpolated average scene luminance across frames and a is the key.

So it seems to me that the process of calculating the average scene luminance Lavg and from there the relative luminance Lr can be shared accross the different tonemapping curves and can happen just right before whatever curve is applied (not just Reinhard). Otherwise I can't see a general way to calculate automatic autoexposure.

See If understand then:

1) Get scene in HDR
2) Go from RGB to CIE Yxy
3) calculate average scene Yavg using Reinhard method (sum of logL div N)
4) calculate relative luminance Yr = (Y(x,y) / Yavg)*a (a is the key)
5) calculate autoexposure considering the average scene luminance of the previous frame Lavg(i-1) and the one of the current frame Lavg(i) and interpolate between them using an exponential falloff for example...
6) use the new adapted luminance in 4) to calculate the relative adapted luminance of the pixel (x,y) ?

so basically Lavg is always the interpolated average luminance across frames ?

7) use the Yr in the tonemapping curve to get Yt (i.e. tonemapped luminance):

if Reinhard:

Yt = Yr / (1+Yr)

if filmic curve (uncharted 2 variant):

Yt = ((Yr*(A*Yr+C*B)+D*E)/(Yr*(A*Yr+B)+D*F))-E/F

and more generally if whatever tonemapping function f(x):

f(Yr) = Yt

8) Transform from Yt xy CIE to RGB to get the tonemapped RGB color given its tonemapped luminance Lt which is calculated from the Yr, wihch is in turn calculated from the interpolated average scene luminance Lavg across frames (automatic exposure control).

Now is this the correct way of calculating autoexposure before applying any tonemapping curve to Yr at all ?

And why someone still apply tonemapping on RGB values straight away knowing that is wrong ? (John Hable said that is not correct to apply the reinhard curve to each rgb channel straight away but at the same time its examples are all doing that, maybe is half correct ?! ) Maybe CIE is more correct but because we can't alpha blend in that space we can live with a less correct solution applying tonemapping on rgb right away. But we have to still interpolate Lavg across frames to have the autoexposure control.

A quite important note: Since I send all my lights in one batch I need to have support for alpha blending, therefore I'm thinking to use CIE Yxy only during post fx and tonemap as being on PC I won't have any problem in fp blending support (as instead happens for other platforms ;) ).
I guess the variant Luv is just convenient if we don't have fp blend support (which on PC is not case anymore by long time). So the idea of accumulating lights in Luv is justified only if the underlying platform doesn't have support for fp render target blending and therefore we are constrained in relying on 8888 unsigned light accumulation buffer right ?

Edited by MegaPixel, 24 September 2012 - 08:02 AM.

### #2allingm  Members

535
Like
1Likes
Like

Posted 24 September 2012 - 11:25 PM

The dot product you are talking about is actually doing the RGB to Yxy conversion, but because all you care about is the luminance it simplifies down to a dot product. However, if you want to properly convert the whole color, RGB, to Yxy and then scale it and then convert it back it is a much more intensive operation. So, both are correct. One contains all the information necessary for RGB -> Yxy -> RGB and one only contains enough for RGB -> Y.

I found the equations here: http://stackoverflow.com/questions/7104034/yxy-to-rgb-conversion If you do the algebra on the conversion functions you'll find what I said to be true.

### #3allingm  Members

535
Like
0Likes
Like

Posted 24 September 2012 - 11:31 PM

I forgot to make this clear, but the Y in Yxy represents luminance.

### #4MJP  Moderators

18204
Like
0Likes
Like

Posted 24 September 2012 - 11:38 PM

It's actually necessary to apply your tone mapping curve to luminance. This is what Reinhard an a lot of other academic papers propose, since it adjusts "intensity" while still maintaining the relative levels of R, G, and B. When simulating film curves its more common to work with RGB (or whatever your spectrum is) directly, and that's what Hable did with his curve. When you're using Reinhard's method for auto-exposure, you just end up with a scalar ratio after you divide the key value by the average luminance which you can then just multiply with your RGB values directly.

### #5MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 01:29 AM

It's actually necessary to apply your tone mapping curve to luminance. This is what Reinhard an a lot of other academic papers propose, since it adjusts "intensity" while still maintaining the relative levels of R, G, and B. When simulating film curves its more common to work with RGB (or whatever your spectrum is) directly, and that's what Hable did with his curve. When you're using Reinhard's method for auto-exposure, you just end up with a scalar ratio after you divide the key value by the average luminance which you can then just multiply with your RGB values directly.

So how you model the autoexposure in the case of a filmic curve if you work solely on RGB and you don't have luminance ?

But it's not correct to multiply RGB with the tonemapped Luminance ... There was a thread on gamedev in which they were converting in Yxy, adjust Y to get Y' and then given Y'xy convert back to RGB, which is different from multiplying RGB straight.

http://www.gamedev.n...r-tone-mapping/

Thanks

Edited by MegaPixel, 25 September 2012 - 03:50 AM.

### #6MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 01:57 AM

I forgot to make this clear, but the Y in Yxy represents luminance.

Yeah I know the meaning of everything I just wanted to understand in the context of different tone mapping curve how to model autoexposure and I was trying to understand if the calculation of the relative luminance as it happens in Reinhard was usable also while varying the tone mapping curve (I mean not just using the Reinhard one but also a filmic for example). Since filmic is working on RGB and not on Luminance.

### #7Ashaman73  Members

13649
Like
0Likes
Like

Posted 25 September 2012 - 05:30 AM

Since filmic is working on RGB and not on Luminance.

Still filmic use an exposure level, or better said, make an assumption about the used exposure level of the rendered image. So why not adjusting this exposure level depending on the average luminance level ?

Ashaman

### #8MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 06:01 AM

Since filmic is working on RGB and not on Luminance.

Still filmic use an exposure level, or better said, make an assumption about the used exposure level of the rendered image. So why not adjusting this exposure level depending on the average luminance level ?

So that means that I can post multiply the RGB color with the adjusted luminance before to run it through the filmic curve or after ?

Also, shouldn't I have to go from CIE to RGB and viceversa and not just post multiply RGB by the adjusted luminance ?

### #9MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 06:49 AM

Just to clarify further I'll post the shader code in which I apply Reinhard (you can see uncharted filmic commented);

float4 PSDeferred(PS_INPUT Input) : SV_Target0{
float4 sceneColor		    = hdrSceneBuffer.Load(int3(Input.screenPos.xy,0));  //get linear hdr color
float4 bloomColor		    = hdrBloomBuffer.Sample(bloomSampler,Input.tc);  //get linear hdr bloom color

//
const float3x3 RGB2XYZ = {0.4124f, 0.3576f, 0.1805f,
0.2126f, 0.7152f, 0.0722f,
0.0193f, 0.1192f, 0.9505f };
//go from RGB color space to CIE
float3 XYZ				   = mul(RGB2XYZ,sceneColor.rgb);
float  den				   = 1.f/dot(float3(1,1,1),XYZ);
float3 Yxy				   = float3(XYZ.y,XYZ.xy*den);

//Reinhard tonemapping operator
float  fragmentLuminance	 = dot(sceneColor.rgb,XYZ.y);
float  relativeLuminance	 = (fragmentLuminance / (averageSceneLuminance+0.0001f))*key;

float  tonemappedLuminance   = (relativeLuminance*(1.f+(relativeLuminance/(Lw*Lw)))) / (1.f+relativeLuminance);
//now go from CIE back to RGB with new tonemapped luminance
const float3x3 XYZ2RGB = {3.2405f, -1.5371f, -0.4985f,
-0.9693f, 1.8760f, 0.0416f,
0.0556f, -0.2040f, 1.0572f };
float s = (tonemappedLuminance/Yxy.z);
XYZ.x   = Yxy.y*s;
XYZ.y   = tonemappedLuminance;
XYZ.z   = (1.f-Yxy.y-Yxy.z)*s;
float4 finalColor		    = float4(mul(XYZ2RGB,XYZ),1);//sceneColor*tonemappedLuminance;
//

//filmic tonemapping (Uncharted2)
/*finalColor		 = max(0.f,finalColor-0.004f);
float4 gammaColor  = (finalColor*(6.2f*finalColor+0.5f))/(finalColor*(6.2f*finalColor+1.7f)+0.06f);
//out the color applying gamma correction
return gammaColor+pow(bloomColor,1.f/2.2f);*/

return pow(finalColor+bloomColor*2.f,1.f/2.2f);


How I can control autoexposure for the filmic one ?

### #10Ashaman73  Members

13649
Like
0Likes
Like

Posted 25 September 2012 - 06:55 AM

So that means that I can post multiply the RGB color with the adjusted luminance before to run it through the filmic curve or after ?

It is the question of what you want to archieve.

If you just want to fake the human eye (adjusting to the darkness), then it would be worth an attempt. According to the filmic tonemapper function here, I would just alter the ExposureBias according to the average luminance. It is more like playing around with the mapper until it looks good vs. a physical correct solution.

Simple example (don't know if it look good):
[s]ExposureBias = 1.5 + (1.0-avgLumi);[/s]


Edit: better solution below from CryZe

Edited by Ashaman73, 25 September 2012 - 11:49 PM.

Ashaman

### #11MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 07:03 AM

So that means that I can post multiply the RGB color with the adjusted luminance before to run it through the filmic curve or after ?

It is the question of what you want to archieve.

If you just want to fake the human eye (adjusting to the darkness), then it would be worth an attempt. According to the filmic tonemapper function here, I would just alter the ExposureBias according to the average luminance. It is more like playing around with the mapper until it looks good vs. a physical correct solution.

so that means that I should use
float  relativeLuminance  = (fragmentLuminance / (averageSceneLuminance+0.0001f))*key;


to modulate the color without run it into the Reinhard tonemapper:

float  tonemappedLuminance   = (relativeLuminance*(1.f+(relativeLuminance/(Lw*Lw)))) / (1.f+relativeLuminance);).


so for the filmic curve should be:

finalColor				*= relativeLuminance;
finalColor				 = max(0.f,finalColor-0.004f);
float4 gammaColor  = (finalColor*(6.2f*finalColor+0.5f))/(finalColor*(6.2f*finalColor+1.7f)+0.06f);
//out the color applying gamma correction
return gammaColor+pow(bloomColor,1.f/2.2f);


(no CIE to RGB needed in this case as the filmic curves works in RGB space straight away)

Edited by MegaPixel, 25 September 2012 - 07:09 AM.

### #12MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 07:12 AM

So that means that I can post multiply the RGB color with the adjusted luminance before to run it through the filmic curve or after ?

It is the question of what you want to archieve.

If you just want to fake the human eye (adjusting to the darkness), then it would be worth an attempt. According to the filmic tonemapper function here, I would just alter the ExposureBias according to the average luminance. It is more like playing around with the mapper until it looks good vs. a physical correct solution.

Simple example (don't know if it look good):
ExposureBias = 1.5 + (1.0-avgLumi);


I understand the idea of tweaking etc. but I'd like first to understand the correct underlying theory and then tweak from there. If you work on the avgLum directly means that you are not using the scaled one L(x,y) / avgLum which is the pixel relative luminance (assume key =1 ) ... see Reinhard for reference

### #13CryZe  Members

773
Like
0Likes
Like

Posted 25 September 2012 - 07:19 AM

I would just alter the ExposureBias according to the average luminance.

Don't do that. The exposure bias is part of Hable's tone mapping operator. It's there to assure that the average luminance (1) gets mapped to 0.5. Otherwise it would be way brighter. You don't need that with other tone mapping operators though.

BTT: I'm not quite sure what you guys are doing. It's this easy:

float3 color = sampleColor(texCoord);
float3 bloomColor = sampleBloom(texCoord);

//Add the bloom (bloomFactor is some constant, 0.0075f is a good value imho)
color += bloomFactor * bloomColor;

//Calculate the exposure (this is the simplest way to do it)
float exposure = 1 / averageLuminance;

float3 color *= exposure;

//Map the color to LDR (Serr tone mapping operator, the same as Hables standard values)
float3 tonemappedColor = 4.6f * color / (3.6f * color + 5.6f);

//Adjust gamma and return (ajusting gamma might not be needed)
return pow(tonemappedColor, 1/2.2f);


Edited by CryZe, 25 September 2012 - 07:22 AM.

### #14MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 07:23 AM

I would just alter the ExposureBias according to the average luminance.

Don't do that. The exposure bias is part of Hable's tone mapping operator. It's there to assure that the average luminance (1) gets mapped to 0.5. Otherwise it would be way brighter. You don't need that with other tone mapping operators though.

BTT: I'm not quite sure what you guys are doing. It's this easy:

float3 color = sampleColor(texCoord);
float3 bloomColor = sampleBloom(texCoord);

//Add the bloom (bloomFactor is some constant, 0.0075f is a good value imho)
color += bloomFactor * bloomColor;

//Calculate the exposure (this is the simplest way to do it)
float exposure = 1 / averageLuminance;

float3 color *= exposure;

//Map the color to LDR (Serr tone mapping operator, the same as Hables standard values)
float3 tonemappedColor = 4.6f * color / (3.6f * color + 5.6f);

//Adjust gamma and return (ajusting gamma might not be needed)
return pow(tonemappedColor, 1/2.2f);


you run the tonemapping process even whem you have to perform the bright pass ? I guess yes since you want to cut the luminance under a given threshold value... ?

### #15MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 07:25 AM

I would just alter the ExposureBias according to the average luminance.

Don't do that. The exposure bias is part of Hable's tone mapping operator. It's there to assure that the average luminance (1) gets mapped to 0.5. Otherwise it would be way brighter. You don't need that with other tone mapping operators though.

BTT: I'm not quite sure what you guys are doing. It's this easy:

float3 color = sampleColor(texCoord);
float3 bloomColor = sampleBloom(texCoord);

//Add the bloom (bloomFactor is some constant, 0.0075f is a good value imho)
color += bloomFactor * bloomColor;

//Calculate the exposure (this is the simplest way to do it)
float exposure = 1 / averageLuminance;

float3 color *= exposure;

//Map the color to LDR (Serr tone mapping operator, the same as Hables standard values)
float3 tonemappedColor = 4.6f * color / (3.6f * color + 5.6f);

//Adjust gamma and return (ajusting gamma might not be needed)
return pow(tonemappedColor, 1/2.2f);


you run the tonemapping process even whem you have to perform the bright pass ? I guess yes since you want to cut the luminance under a given threshold value... ?
Also can you point me to some good reading to understand first the simplest way instead of just applying Reinhard or whatever other tonemapper blindly ? thanks in advance for your help ;)
Btw I tried your serr approach and my scene looked way too much bright and I have just one pointlight !

Edited by MegaPixel, 25 September 2012 - 07:32 AM.

### #16CryZe  Members

773
Like
0Likes
Like

Posted 25 September 2012 - 07:31 AM

you run the tonemapping process even whem you have to perform the bright pass ? I guess yes since you want to cut the luminance under a given threshold value... ?

There's no need for a bright pass with a threshold value if you actually have high dynamic range values. Threshold values for bloom are from back in the days where you couldn't afford the bandwidth for HDR illumination. Bright lights are thousands if not millions of times brighter than the rest of your scene (or at least they should ). By simply using a multiplicative factor you reduce the amount of bloom so that only the bloom of these really bright lights is visible. So there's no need for either a bright pass or a threshold value for bloom. A threshold value is just not realistic. Lenses always scatter a tiny percentage of the light onto nearby pixels (depending on the quality of the lenses), they never subtract light.

Edited by CryZe, 25 September 2012 - 07:35 AM.

### #17MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 07:35 AM

you run the tonemapping process even whem you have to perform the bright pass ? I guess yes since you want to cut the luminance under a given threshold value... ?

There's no need for a bright pass with a threshold value if you actually have high dynamic range values. Threshold values for bloom are from back in the days where you couldn't afford the bandwidth for HDR illumination. Bright lights are thousands if not millions of times brighter than the rest of your scene (or at least they should ). By simply using a multiplicative factor you reduce the amount of bloom so that only the bloom of these really bright lights is visible. So there's no need for either a bright pass or a threshold value for bloom.

So you scale down the color intensity by some factor before gaussian blur is applied (like during the downsampling of the scene something like sceneColor*threshold)? But it doesn't seem to be effective, because what I actually want is for only some part of the scene to glow and I don't want to see all the scene blurred ...

Edited by MegaPixel, 25 September 2012 - 07:38 AM.

### #18CryZe  Members

773
Like
0Likes
Like

Posted 25 September 2012 - 07:38 AM

It doesn't matter whether you do it before or afterwards. I do it when combining it with the source color again. But the result is the same either way. Not everything will be blurred. Like I said, with proper high dynamic range, it won't happen. Simply decrease the factor to something where not everything is glowing and increase the luminance of everything you want to see glowing (you might need to make them thousands of times brighter, but this is actually realistic). And that's why you need the tone mapper. The tone mapper will reduce their perceived luminance and lets them look like they wouldn't actually be thousands of times brighter.

The sun in my engine is for example 6.5 million lux while shadows on a sunny day are just 10,000 lux, nights are 0.005 lux.

Edited by CryZe, 25 September 2012 - 07:58 AM.

### #19MegaPixel  Members

241
Like
0Likes
Like

Posted 25 September 2012 - 08:20 AM

It doesn't matter whether you do it before or afterwards. I do it when combining it with the source color again. But the result is the same either way. Not everything will be blurred. Like I said, with proper high dynamic range, it won't happen. Simply decrease the factor to something where not everything is glowing and increase the luminance of everything you want to see glowing (you might need to make them thousands of times brighter, but this is actually realistic). And that's why you need the tone mapper. The tone mapper will reduce their perceived luminance and lets them look like they wouldn't actually be thousands of times brighter.

The sun in my engine is for example 6.5 million lux while shadows on a sunny day are just 10,000 lux, nights are 0.005 lux.

How you define a lux in your engine ?

Also to have good bloom I was thinking to downsample 5 times, something like:

1) blur while downsampling the scene for the first time
2) blur while downsampling to the next level
...
till 5) ...

then:

upsample each downscaled version of the previous 5 steps JUST to the previous level (so don't stretch a given level to fullscreen?) looks like Epic is doing like this in unreal engine 4 ...
I just have the doubt that they actually stretch each level to fullscreen while adding them together ... can you confirm on that ?

Old topic!

Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.