• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
MegaPixel

Multiply RGB by luminance or working in CIE ? Which is more correct ?

18 posts in this topic

Hi guys,

I was studying through tonemapping operators and exposure control.

I've read several threads on gamedev about those topics and I've also read the MJP article on tonemapping on his blog and John Hable on filmic S curve.

Just one thing Is not clear to me: I understand that exposure control and tonemapping are two different things, but I wonder why someone is still multipling the RGB color by the adapted/tonemapped luminance L straight away.
Shouldn't be more correct to go in CIE color space Yxy, adapt Y to get Y' and then given Y'xy go back to RGB to get the new tonemapped color ?
Also, given that automatic exposure is a different thing with respect to tonemapping (which brings the Luminance values in [0,1] range in Reinhard) why in the Reinhard paper he basically calculates the geometric mean to get the average luminance in the context of a tonemapping process (I mean it's just part of the process but is still not the tonemapping step), that means that the relative luminance then can be used in the context of any tonemapping curve (not just reinhard) to have automatic exposure control ? And therefore shouldn't the relative luminance Lr calulation (the one before Lr / (1 + Lr), which is the actual tonemapped Lt ) consider the interpolated Lavg across frames in its calculation ?

Lr = (L(x,y) / Lavg)*a, where Lavg is the interpolated average scene luminance across frames and a is the key.

So it seems to me that the process of calculating the average scene luminance Lavg and from there the relative luminance Lr can be shared accross the different tonemapping curves and can happen just right before whatever curve is applied (not just Reinhard). Otherwise I can't see a general way to calculate automatic autoexposure.


See If understand then:

1) Get scene in HDR
2) Go from RGB to CIE Yxy
3) calculate average scene Yavg using Reinhard method (sum of logL div N)
4) calculate relative luminance Yr = (Y(x,y) / Yavg)*a (a is the key)
5) calculate autoexposure considering the average scene luminance of the previous frame Lavg(i-1) and the one of the current frame Lavg(i) and interpolate between them using an exponential falloff for example...
6) use the new adapted luminance in 4) to calculate the relative adapted luminance of the pixel (x,y) ?

so basically Lavg is always the interpolated average luminance across frames ?

7) use the Yr in the tonemapping curve to get Yt (i.e. tonemapped luminance):

if Reinhard:

Yt = Yr / (1+Yr)

if filmic curve (uncharted 2 variant):

Yt = ((Yr*(A*Yr+C*B)+D*E)/(Yr*(A*Yr+B)+D*F))-E/F

and more generally if whatever tonemapping function f(x):

f(Yr) = Yt

8) Transform from Yt xy CIE to RGB to get the tonemapped RGB color given its tonemapped luminance Lt which is calculated from the Yr, wihch is in turn calculated from the interpolated average scene luminance Lavg across frames (automatic exposure control).

Pfu, I made it.

Now is this the correct way of calculating autoexposure before applying any tonemapping curve to Yr at all ?

And why someone still apply tonemapping on RGB values straight away knowing that is wrong ? (John Hable said that is not correct to apply the reinhard curve to each rgb channel straight away but at the same time its examples are all doing that, maybe is half correct ?! ) Maybe CIE is more correct but because we can't alpha blend in that space we can live with a less correct solution applying tonemapping on rgb right away. But we have to still interpolate Lavg across frames to have the autoexposure control.

A quite important note: Since I send all my lights in one batch I need to have support for alpha blending, therefore I'm thinking to use CIE Yxy only during post fx and tonemap as being on PC I won't have any problem in fp blending support (as instead happens for other platforms ;) ).
I guess the variant Luv is just convenient if we don't have fp blend support (which on PC is not case anymore by long time). So the idea of accumulating lights in Luv is justified only if the underlying platform doesn't have support for fp render target blending and therefore we are constrained in relying on 8888 unsigned light accumulation buffer right ?

Thanks in advance for any reply Edited by MegaPixel
1

Share this post


Link to post
Share on other sites
The dot product you are talking about is actually doing the RGB to Yxy conversion, but because all you care about is the luminance it simplifies down to a dot product. However, if you want to properly convert the whole color, RGB, to Yxy and then scale it and then convert it back it is a much more intensive operation. So, both are correct. One contains all the information necessary for RGB -> Yxy -> RGB and one only contains enough for RGB -> Y.

I found the equations here: http://stackoverflow.com/questions/7104034/yxy-to-rgb-conversion If you do the algebra on the conversion functions you'll find what I said to be true.
1

Share this post


Link to post
Share on other sites
It's actually necessary to apply your tone mapping curve to luminance. This is what Reinhard an a lot of other academic papers propose, since it adjusts "intensity" while still maintaining the relative levels of R, G, and B. When simulating film curves its more common to work with RGB (or whatever your spectrum is) directly, and that's what Hable did with his curve. When you're using Reinhard's method for auto-exposure, you just end up with a scalar ratio after you divide the key value by the average luminance which you can then just multiply with your RGB values directly.
0

Share this post


Link to post
Share on other sites
[quote name='MJP' timestamp='1348551497' post='4983466']
It's actually necessary to apply your tone mapping curve to luminance. This is what Reinhard an a lot of other academic papers propose, since it adjusts "intensity" while still maintaining the relative levels of R, G, and B. When simulating film curves its more common to work with RGB (or whatever your spectrum is) directly, and that's what Hable did with his curve. When you're using Reinhard's method for auto-exposure, you just end up with a scalar ratio after you divide the key value by the average luminance which you can then just multiply with your RGB values directly.
[/quote]

So how you model the autoexposure in the case of a filmic curve if you work solely on RGB and you don't have luminance ?

But it's not correct to multiply RGB with the tonemapped Luminance ... There was a thread on gamedev in which they were converting in Yxy, adjust Y to get Y' and then given Y'xy convert back to RGB, which is different from multiplying RGB straight.

Here is the thread:
[url="http://www.gamedev.net/topic/484641-which-color-space-for-tone-mapping/"]http://www.gamedev.n...r-tone-mapping/[/url]

Thanks Edited by MegaPixel
0

Share this post


Link to post
Share on other sites
[quote name='allingm' timestamp='1348551068' post='4983462']
I forgot to make this clear, but the Y in Yxy represents luminance.
[/quote]

Yeah I know the meaning of everything I just wanted to understand in the context of different tone mapping curve how to model autoexposure and I was trying to understand if the calculation of the relative luminance as it happens in Reinhard was usable also while varying the tone mapping curve (I mean not just using the Reinhard one but also a filmic for example). Since filmic is working on RGB and not on Luminance.
0

Share this post


Link to post
Share on other sites
[quote name='MegaPixel' timestamp='1348559831' post='4983501']
Since filmic is working on RGB and not on Luminance.
[/quote]
Still filmic use an exposure level, or better said, make an assumption about the used exposure level of the rendered image. So why not adjusting this exposure level depending on the average luminance level ?
0

Share this post


Link to post
Share on other sites
[quote name='Ashaman73' timestamp='1348572607' post='4983527']
[quote name='MegaPixel' timestamp='1348559831' post='4983501']
Since filmic is working on RGB and not on Luminance.
[/quote]
Still filmic use an exposure level, or better said, make an assumption about the used exposure level of the rendered image. So why not adjusting this exposure level depending on the average luminance level ?
[/quote]

So that means that I can post multiply the RGB color with the adjusted luminance before to run it through the filmic curve or after ?

Also, shouldn't I have to go from CIE to RGB and viceversa and not just post multiply RGB by the adjusted luminance ?
0

Share this post


Link to post
Share on other sites
Just to clarify further I'll post the shader code in which I apply Reinhard (you can see uncharted filmic commented);

[CODE]
float4 PSDeferred(PS_INPUT Input) : SV_Target0{
float4 sceneColor = hdrSceneBuffer.Load(int3(Input.screenPos.xy,0)); //get linear hdr color
float4 bloomColor = hdrBloomBuffer.Sample(bloomSampler,Input.tc); //get linear hdr bloom color

//
const float3x3 RGB2XYZ = {0.4124f, 0.3576f, 0.1805f,
0.2126f, 0.7152f, 0.0722f,
0.0193f, 0.1192f, 0.9505f };
//go from RGB color space to CIE
float3 XYZ = mul(RGB2XYZ,sceneColor.rgb);
float den = 1.f/dot(float3(1,1,1),XYZ);
float3 Yxy = float3(XYZ.y,XYZ.xy*den);

//Reinhard tonemapping operator
float averageSceneLuminance = exp(luminanceBuffer.Load(int3(0,0,9)).r);
float fragmentLuminance = dot(sceneColor.rgb,XYZ.y);
float relativeLuminance = (fragmentLuminance / (averageSceneLuminance+0.0001f))*key;

float tonemappedLuminance = (relativeLuminance*(1.f+(relativeLuminance/(Lw*Lw)))) / (1.f+relativeLuminance);
//now go from CIE back to RGB with new tonemapped luminance
const float3x3 XYZ2RGB = {3.2405f, -1.5371f, -0.4985f,
-0.9693f, 1.8760f, 0.0416f,
0.0556f, -0.2040f, 1.0572f };
float s = (tonemappedLuminance/Yxy.z);
XYZ.x = Yxy.y*s;
XYZ.y = tonemappedLuminance;
XYZ.z = (1.f-Yxy.y-Yxy.z)*s;
float4 finalColor = float4(mul(XYZ2RGB,XYZ),1);//sceneColor*tonemappedLuminance;
//

//filmic tonemapping (Uncharted2)
/*finalColor = max(0.f,finalColor-0.004f);
float4 gammaColor = (finalColor*(6.2f*finalColor+0.5f))/(finalColor*(6.2f*finalColor+1.7f)+0.06f);
//out the color applying gamma correction
return gammaColor+pow(bloomColor,1.f/2.2f);*/



return pow(finalColor+bloomColor*2.f,1.f/2.2f);
[/CODE]

How I can control autoexposure for the filmic one ?
0

Share this post


Link to post
Share on other sites
[quote name='MegaPixel' timestamp='1348574474' post='4983535']
So that means that I can post multiply the RGB color with the adjusted luminance before to run it through the filmic curve or after ?
[/quote]
It is the question of what you want to archieve.

If you just want to fake the human eye (adjusting to the darkness), then it would be worth an attempt. According to the filmic tonemapper function [url="http://filmicgames.com/archives/75"]here[/url],[s] I would just alter the ExposureBias according to the average luminance[/s]. It is more like playing around with the mapper until it looks good vs. a physical correct solution.

[s]Simple example (don't know if it look good):[/s]
[CODE]
[s]ExposureBias = 1.5 + (1.0-avgLumi);[/s]
[/CODE]

Edit: better solution below from CryZe Edited by Ashaman73
0

Share this post


Link to post
Share on other sites
[quote name='Ashaman73' timestamp='1348577752' post='4983548']
[quote name='MegaPixel' timestamp='1348574474' post='4983535']
So that means that I can post multiply the RGB color with the adjusted luminance before to run it through the filmic curve or after ?
[/quote]
It is the question of what you want to archieve.

If you just want to fake the human eye (adjusting to the darkness), then it would be worth an attempt. According to the filmic tonemapper function [url="http://filmicgames.com/archives/75"]here[/url], I would just alter the ExposureBias according to the average luminance. It is more like playing around with the mapper until it looks good vs. a physical correct solution.
[/quote]

so that means that I should use
[CODE]
float relativeLuminance = (fragmentLuminance / (averageSceneLuminance+0.0001f))*key;
[/CODE]

to modulate the color without run it into the Reinhard tonemapper:

[CODE]
float tonemappedLuminance = (relativeLuminance*(1.f+(relativeLuminance/(Lw*Lw)))) / (1.f+relativeLuminance);).
[/CODE]

so for the filmic curve should be:

[CODE]
finalColor *= relativeLuminance;
finalColor = max(0.f,finalColor-0.004f);
float4 gammaColor = (finalColor*(6.2f*finalColor+0.5f))/(finalColor*(6.2f*finalColor+1.7f)+0.06f);
//out the color applying gamma correction
return gammaColor+pow(bloomColor,1.f/2.2f);
[/CODE]

(no CIE to RGB needed in this case as the filmic curves works in RGB space straight away) Edited by MegaPixel
0

Share this post


Link to post
Share on other sites
[quote name='Ashaman73' timestamp='1348577752' post='4983548']
[quote name='MegaPixel' timestamp='1348574474' post='4983535']
So that means that I can post multiply the RGB color with the adjusted luminance before to run it through the filmic curve or after ?
[/quote]
It is the question of what you want to archieve.

If you just want to fake the human eye (adjusting to the darkness), then it would be worth an attempt. According to the filmic tonemapper function [url="http://filmicgames.com/archives/75"]here[/url], I would just alter the ExposureBias according to the average luminance. It is more like playing around with the mapper until it looks good vs. a physical correct solution.

Simple example (don't know if it look good):
[CODE]
ExposureBias = 1.5 + (1.0-avgLumi);
[/CODE]
[/quote]

I understand the idea of tweaking etc. but I'd like first to understand the correct underlying theory and then tweak from there. If you work on the avgLum directly means that you are not using the scaled one L(x,y) / avgLum which is the pixel relative luminance (assume key =1 ) ... see Reinhard for reference
0

Share this post


Link to post
Share on other sites
[quote name='Ashaman73' timestamp='1348577752' post='4983548']
I would just alter the ExposureBias according to the average luminance.
[/quote]
Don't do that. The exposure bias is part of Hable's tone mapping operator. It's there to assure that the average luminance (1) gets mapped to 0.5. Otherwise it would be way brighter. You don't need that with other tone mapping operators though.

BTT: I'm not quite sure what you guys are doing. It's this easy:

[CODE]
float3 color = sampleColor(texCoord);
float3 bloomColor = sampleBloom(texCoord);

//Add the bloom (bloomFactor is some constant, 0.0075f is a good value imho)
color += bloomFactor * bloomColor;

//Calculate the exposure (this is the simplest way to do it)
float exposure = 1 / averageLuminance;

//Adjust the exposure
float3 color *= exposure;

//Map the color to LDR (Serr tone mapping operator, the same as Hables standard values)
float3 tonemappedColor = 4.6f * color / (3.6f * color + 5.6f);

//Adjust gamma and return (ajusting gamma might not be needed)
return pow(tonemappedColor, 1/2.2f);
[/CODE] Edited by CryZe
0

Share this post


Link to post
Share on other sites
[quote name='CryZe' timestamp='1348579184' post='4983559']
[quote name='Ashaman73' timestamp='1348577752' post='4983548']
I would just alter the ExposureBias according to the average luminance.
[/quote]
Don't do that. The exposure bias is part of Hable's tone mapping operator. It's there to assure that the average luminance (1) gets mapped to 0.5. Otherwise it would be way brighter. You don't need that with other tone mapping operators though.

BTT: I'm not quite sure what you guys are doing. It's this easy:

[CODE]
float3 color = sampleColor(texCoord);
float3 bloomColor = sampleBloom(texCoord);

//Add the bloom (bloomFactor is some constant, 0.0075f is a good value imho)
color += bloomFactor * bloomColor;

//Calculate the exposure (this is the simplest way to do it)
float exposure = 1 / averageLuminance;

//Adjust the exposure
float3 color *= exposure;

//Map the color to LDR (Serr tone mapping operator, the same as Hables standard values)
float3 tonemappedColor = 4.6f * color / (3.6f * color + 5.6f);

//Adjust gamma and return (ajusting gamma might not be needed)
return pow(tonemappedColor, 1/2.2f);
[/CODE]
[/quote]

you run the tonemapping process even whem you have to perform the bright pass ? I guess yes since you want to cut the luminance under a given threshold value... ?
0

Share this post


Link to post
Share on other sites
[quote name='MegaPixel' timestamp='1348579435' post='4983562']
[quote name='CryZe' timestamp='1348579184' post='4983559']
[quote name='Ashaman73' timestamp='1348577752' post='4983548']
I would just alter the ExposureBias according to the average luminance.
[/quote]
Don't do that. The exposure bias is part of Hable's tone mapping operator. It's there to assure that the average luminance (1) gets mapped to 0.5. Otherwise it would be way brighter. You don't need that with other tone mapping operators though.

BTT: I'm not quite sure what you guys are doing. It's this easy:

[CODE]
float3 color = sampleColor(texCoord);
float3 bloomColor = sampleBloom(texCoord);

//Add the bloom (bloomFactor is some constant, 0.0075f is a good value imho)
color += bloomFactor * bloomColor;

//Calculate the exposure (this is the simplest way to do it)
float exposure = 1 / averageLuminance;

//Adjust the exposure
float3 color *= exposure;

//Map the color to LDR (Serr tone mapping operator, the same as Hables standard values)
float3 tonemappedColor = 4.6f * color / (3.6f * color + 5.6f);

//Adjust gamma and return (ajusting gamma might not be needed)
return pow(tonemappedColor, 1/2.2f);
[/CODE]
[/quote]
[/quote]

you run the tonemapping process even whem you have to perform the bright pass ? I guess yes since you want to cut the luminance under a given threshold value... ?
Also can you point me to some good reading to understand first the simplest way instead of just applying Reinhard or whatever other tonemapper blindly ? thanks in advance for your help ;)
Btw I tried your serr approach and my scene looked way too much bright and I have just one pointlight ! Edited by MegaPixel
0

Share this post


Link to post
Share on other sites
[quote name='MegaPixel' timestamp='1348579435' post='4983562']
you run the tonemapping process even whem you have to perform the bright pass ? I guess yes since you want to cut the luminance under a given threshold value... ?
[/quote]
There's no need for a bright pass with a threshold value if you actually have high dynamic range values. Threshold values for bloom are from back in the days where you couldn't afford the bandwidth for HDR illumination. Bright lights are thousands if not millions of times brighter than the rest of your scene (or at least they should [img]http://public.gamedev.net//public/style_emoticons/default/biggrin.png[/img]). By simply using a multiplicative factor you reduce the amount of bloom so that only the bloom of these really bright lights is visible. So there's no need for either a bright pass or a threshold value for bloom. A threshold value is just not realistic. Lenses always scatter a tiny percentage of the light onto nearby pixels (depending on the quality of the lenses), they never subtract light. Edited by CryZe
0

Share this post


Link to post
Share on other sites
[quote name='CryZe' timestamp='1348579897' post='4983568']
[quote name='MegaPixel' timestamp='1348579435' post='4983562']
you run the tonemapping process even whem you have to perform the bright pass ? I guess yes since you want to cut the luminance under a given threshold value... ?
[/quote]
There's no need for a bright pass with a threshold value if you actually have high dynamic range values. Threshold values for bloom are from back in the days where you couldn't afford the bandwidth for HDR illumination. Bright lights are thousands if not millions of times brighter than the rest of your scene (or at least they should [img]http://public.gamedev.net//public/style_emoticons/default/biggrin.png[/img]). By simply using a multiplicative factor you reduce the amount of bloom so that only the bloom of these really bright lights is visible. So there's no need for either a bright pass or a threshold value for bloom.
[/quote]

So you scale down the color intensity by some factor before gaussian blur is applied (like during the downsampling of the scene something like sceneColor*threshold)? But it doesn't seem to be effective, because what I actually want is for only some part of the scene to glow and I don't want to see all the scene blurred ... Edited by MegaPixel
0

Share this post


Link to post
Share on other sites
It doesn't matter whether you do it before or afterwards. I do it when combining it with the source color again. But the result is the same either way. Not everything will be blurred. Like I said, with proper high dynamic range, it won't happen. Simply decrease the factor to something where not everything is glowing and increase the luminance of everything you want to see glowing (you might need to make them thousands of times brighter, but this is actually realistic). And that's why you need the tone mapper. The tone mapper will reduce their perceived luminance and lets them look like they wouldn't actually be thousands of times brighter.

The sun in my engine is for example 6.5 million lux while shadows on a sunny day are just 10,000 lux, nights are 0.005 lux. Edited by CryZe
0

Share this post


Link to post
Share on other sites
[quote name='CryZe' timestamp='1348580281' post='4983573']
It doesn't matter whether you do it before or afterwards. I do it when combining it with the source color again. But the result is the same either way. Not everything will be blurred. Like I said, with proper high dynamic range, it won't happen. Simply decrease the factor to something where not everything is glowing and increase the luminance of everything you want to see glowing (you might need to make them thousands of times brighter, but this is actually realistic). And that's why you need the tone mapper. The tone mapper will reduce their perceived luminance and lets them look like they wouldn't actually be thousands of times brighter.

The sun in my engine is for example 6.5 million lux while shadows on a sunny day are just 10,000 lux, nights are 0.005 lux.
[/quote]

How you define a lux in your engine ?

Also to have good bloom I was thinking to downsample 5 times, something like:

1) blur while downsampling the scene for the first time
2) blur while downsampling to the next level
...
till 5) ...

then:

upsample each downscaled version of the previous 5 steps JUST to the previous level (so don't stretch a given level to fullscreen?) looks like Epic is doing like this in unreal engine 4 ...
I just have the doubt that they actually stretch each level to fullscreen while adding them together ... can you confirm on that ?
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0