Jump to content
  • Advertisement
Sign in to follow this  
MrOMGWTF

Extracting surface reflectance or lightning intensity from a single image.

This topic is 1434 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm was struggling with this for a while. Simply said, I want to approximate the surface reflectance factor for each pixel from a single image. Let's have a look at this simple equation:

1MOam20.gif

Where:

pi - Pixel intensity

sr - Surface reflectance at the current pixel

n, l - Self explanatory

 

Now, mathematically, it's not possible to recover sr just by having pi. So the only option here is to somehow statistically approximate it. How can I accomplish that? I mean, our brain does it everyday, so it's definitely possible to approximate it accurately. Does anybody have any ideas?

Share this post


Link to post
Share on other sites
Advertisement

The surface normal can be retrieved from either a depth texture (if available) or approximated by gradient methods. The light direction will have to be inferred from surrounding context (specular lobes, shadows, are all good indicators of the direction of the light source - if you are outside, know your position, view direction, and the time of day/year, latitude and longitude, you can determine the Sun's position relative to you and work out the sunlight's direction). Once you have all that, you just calculate the dot product and solve for the reflectance. But getting the normal and light direction mechanically will probably require some serious computer vision algorithms if you want to automate it somehow.

 

Not to mention that you are completely ignoring the light's intensity (dependent on the light itself, and on its distance from the surface if it's not too far away). And, of course, this is valid only for a single light source. If you have multiple light sources, things get tricky.

 

Do you have examples of images you need to work on?

Share this post


Link to post
Share on other sites

The surface normal can be retrieved from either a depth texture (if available) or approximated by gradient methods. The light direction will have to be inferred from surrounding context (specular lobes, shadows, are all good indicators of the direction of the light source - if you are outside, know your position, view direction, and the time of day/year, latitude and longitude, you can determine the Sun's position relative to you and work out the sunlight's direction). Once you have all that, you just calculate the dot product and solve for the reflectance. But getting the normal and light direction mechanically will probably require some serious computer vision algorithms if you want to automate it somehow.

 

Not to mention that you are completely ignoring the light's intensity (dependent on the light itself, and on its distance from the surface if it's not too far away). And, of course, this is valid only for a single light source. If you have multiple light sources, things get tricky.

 

Do you have examples of images you need to work on?

Well, the images I'm working on are illuminated by ambient lightning (most likely sky), so no shadows, and the lightning is really smooth. Here is a good example of an image I'd be working on:

3551573740_64c3e2d87f_b.jpg

 

When trying to approximate depth, the dark-is-deep approach doesn't work very well because of the variations in the surface reflectance value. That's why I'm trying to find the surface reflectance value. Diving the above image by its albedo part would result in pure lightning information.

Edited by MrOMGWTF

Share this post


Link to post
Share on other sites
Your example image has a lot of texture and next to no specularity/reflections. Those are ideal conditions for 3D reconstructions from multiple images. I have been experimenting with them for the last couple of months and the results are actually quite nice. I have yet to see a singe-image reconstruction that comes even close to the quality of multi-image reconstructions.

Share this post


Link to post
Share on other sites

So I was playing around with this normal vector reconstruction from a single image, and I must say I'm getting pretty impressive results.
Here's a normal map generated from the image from my previous post:
IDVLm2F.jpg

Another example of a generated normal map:

9zn2LEn.jpg

What do you think?

Edited by MrOMGWTF

Share this post


Link to post
Share on other sites

This looks quite convincing, but naturally there are still some artefacts related to what you called "dark-is-deep" -- white spots are given little hills that obviously aren't really there. A lot of the instances I've seen of this seem to roughly speaking construct a depth map from a sum of blurred layers of the image (often with what seems like some extra hand-wavy convolution applied at each step), and it looks like you're perhaps doing something similar?
 

I suspect a purely statistical approach based on windows of pixels might actually work pretty well, but I'm not sure if there's enough easily-available reference data to generate a model.

Share this post


Link to post
Share on other sites

This looks quite convincing, but naturally there are still some artefacts related to what you called "dark-is-deep" -- white spots are given little hills that obviously aren't really there. A lot of the instances I've seen of this seem to roughly speaking construct a depth map from a sum of blurred layers of the image (often with what seems like some extra hand-wavy convolution applied at each step), and it looks like you're perhaps doing something similar?
 

I suspect a purely statistical approach based on windows of pixels might actually work pretty well, but I'm not sure if there's enough easily-available reference data to generate a model.

Yeah it's basically creating a pyramid of blurred greyscale input images, calculating normal maps from them (while treating them like casual depth maps), and adding them together while using some particular weight coefficients. They vary per image. But most of the time the lowest frequency band has the highest weight and it goes down, something like 1/f but not as sudden. While it works pretty good on most images, it doesn't work that good on images with sharp edges. Here's a good example:

3kRKe0l.jpg

As you can see, the cracked concrete at the top has obvious sharp edges, while my algorithm generates smooth edges. I'll be working on it. 

Share this post


Link to post
Share on other sites

This looks quite convincing, but naturally there are still some artefacts related to what you called "dark-is-deep" -- white spots are given little hills that obviously aren't really there. A lot of the instances I've seen of this seem to roughly speaking construct a depth map from a sum of blurred layers of the image (often with what seems like some extra hand-wavy convolution applied at each step), and it looks like you're perhaps doing something similar?
 

I suspect a purely statistical approach based on windows of pixels might actually work pretty well, but I'm not sure if there's enough easily-available reference data to generate a model.

Also, I forgot to mention something. You were talking about how white spots are given little hills. I know it happens, and it happens because of the variation in the albedo of the surface. But that happens only if you work on a simple unprocessed image. If you were to work only on the lightning data, it would work pretty good. (It naturally happens that something that's deeper is darker, ambient occlusion etc, if you do some crazy math here you can actually recover the depth information pretty accurately)

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!