Jump to content
  • Advertisement
Sign in to follow this  
azer.darkblade

Estimating surface normal from depth map and smoothing

This topic is 2124 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi,
Currently i'm doing some image processing on depth map (using OpenNI and XTion)
My depth map size is 320x240 (cannot make it 640x480 due to performance issue since we're developing a game)
The image processing should be real-time so it's fully written in fragment shader.
I generate normal map using simple cross product of 8 neighbouring pixels

normal_estimation_s.png

Since the depth is noisy, i do a hack, so i calculate the 8 pixels like this

[source lang="cpp"]float step = _range/512.0; // 512.0 is tex size
float4 top = tex2D(_SubTex, float2(i.uv.x, i.uv.y - step));
float4 right = tex2D(_SubTex, float2(i.uv.x + step, i.uv.y));
//and so on....[/source]

i can adjust the _range variable and looking for farther pixels

Here's the result (raw depth map, now prepocessing at all)
005.png


2nd result (5x5 bilateral filter is applied first)

004.png

My approach to estimate normal is kinda stupid and i need suggestions to do it in a better way tongue.png
And also i'm considering to apply 5x5 median filter (i've tried 3x3 media filter but it's too small to remove noise) but it will be difficult to use such a huge 5x5 window due to implementation and performance issue Edited by azer.darkblade

Share this post


Link to post
Share on other sites
Advertisement
Yeah, I think picking an arbitrary range and doing a cross product is a little hokey. wink.png How about calculating distance-weighted depths for each of the four corners used? This could be done by only sampling on the diagonals, or alternately by sampling whole quadrants.You may also want to pre-process using a median filter for spot removal.

A more accurate approach might be something like the below, although probably too slow for what you want:
http://www.pointclouds.org/documentation/tutorials/normal_estimation.php

Share this post


Link to post
Share on other sites
This may be fairly random, but I think you and I are working on the same project :) I've been doing the Kinect graphics for the 360 on a strikingly similar background ;)

The approach I used to get fairly good results was to just do a simple downsample to a half size target, leveraging bi-linear filtering. If you need to maintain the size of your normal map as you have it now, this may not work, since it thins out the appendages, but to our 120x90 image it looks great. I also had some really great results by applying a gaussian blur on top of everything.

Share this post


Link to post
Share on other sites

This may be fairly random, but I think you and I are working on the same project smile.png I've been doing the Kinect graphics for the 360 on a strikingly similar background ;)

The approach I used to get fairly good results was to just do a simple downsample to a half size target, leveraging bi-linear filtering. If you need to maintain the size of your normal map as you have it now, this may not work, since it thins out the appendages, but to our 120x90 image it looks great. I also had some really great results by applying a gaussian blur on top of everything.


Hahaha I see biggrin.png , good luck with your 360 project. mine uses Unity
i did a little bit tweak on my bilateral filter and it looks better right now. yes you are right by downsampling and applying gaussian filter, the result is more consistent and less noise, right now i choose not to lose the detail and keep on 320x240 resolution instead. I got depth map and normal map, hey i can add SSAO effect now biggrin.png Edited by azer.darkblade

Share this post


Link to post
Share on other sites
It's me again,
Actually i still have one more issue. As you can see in the image below. My hand is in front of my chest but as as viewed on the normal calculation result the hand seems connected with my chest.

normal.png

Share this post


Link to post
Share on other sites
I see that as a risk of applying general blurring. Personally I would selectively blur, e.g. do spot removal first and then only blur together depth values that are within a certain tolerance of each other. Realistically noise will usually only cause small errors, apart from weird outliers that spot removal could catch.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!