thanks for the replies since it's almost 2 weeks and nobody replied.
i tried hqx but it seems not very good while the image has noise, and so microsoft paper it's not real-time
Anyway, i (almost) solved the problem by using combination of edge detection and spline curve:
here's the result:
The result is really good but the fps dropped to 20-30 fps. The problem is i need a method that runs fast enough in real-time
Another idea is using morphological opening (still thinking how to implement this on shader):
but this one only removes a little noise around the edge, though.
Actually i don't need super-noise-removal algorithm, what i want is just approximation of the shape and loss of detail on several parts is fine to me since the goal is making a silhouette like this one: