Hi,
Currently i'm doing some image processing on depth map (using OpenNI and XTion)
My depth map size is 320x240 (cannot make it 640x480 due to performance issue since we're developing a game)
The image processing should be real-time so it's fully written in fragment shader.
I generate normal map using simple cross product of 8 neighbouring pixels



Since the depth is noisy, i do a hack, so i calculate the 8 pixels like this

[source lang="cpp"]float step = _range/512.0; // 512.0 is tex size
float4 top = tex2D(_SubTex, float2(i.uv.x, i.uv.y - step));
float4 right = tex2D(_SubTex, float2(i.uv.x + step, i.uv.y));
//and so on....[/source]

i can adjust the _range variable and looking for farther pixels

Here's the result (raw depth map, now prepocessing at all)



2nd result (5x5 bilateral filter is applied first)



My approach to estimate normal is kinda stupid and i need suggestions to do it in a better way
And also i'm considering to apply 5x5 median filter (i've tried 3x3 media filter but it's too small to remove noise) but it will be difficult to use such a huge 5x5 window due to implementation and performance issue

Yeah, I think picking an arbitrary range and doing a cross product is a little hokey. How about calculating distance-weighted depths for each of the four corners used? This could be done by only sampling on the diagonals, or alternately by sampling whole quadrants.You may also want to pre-process using a median filter for spot removal.

A more accurate approach might be something like the below, although probably too slow for what you want:
http://www.pointclouds.org/documentation/tutorials/normal_estimation.php

This may be fairly random, but I think you and I are working on the same project I've been doing the Kinect graphics for the 360 on a strikingly similar background ;)

The approach I used to get fairly good results was to just do a simple downsample to a half size target, leveraging bi-linear filtering. If you need to maintain the size of your normal map as you have it now, this may not work, since it thins out the appendages, but to our 120x90 image it looks great. I also had some really great results by applying a gaussian blur on top of everything.

Hahaha I see , good luck with your 360 project. mine uses Unity
i did a little bit tweak on my bilateral filter and it looks better right now. yes you are right by downsampling and applying gaussian filter, the result is more consistent and less noise, right now i choose not to lose the detail and keep on 320x240 resolution instead. I got depth map and normal map, hey i can add SSAO effect now

It's me again,
Actually i still have one more issue. As you can see in the image below. My hand is in front of my chest but as as viewed on the normal calculation result the hand seems connected with my chest.

I see that as a risk of applying general blurring. Personally I would selectively blur, e.g. do spot removal first and then only blur together depth values that are within a certain tolerance of each other. Realistically noise will usually only cause small errors, apart from weird outliers that spot removal could catch.