Hi I am currently working on a piece of software based in WPF 3D(cant use XNA or DirectX). It requires that I take a mesh and deform it using a kinect skeleton. The software has two areas, one where binding is performed and the other where deformation is performed. The basic idea is I take a snapshot of someones skeleton in an approriate bind pose. I position the mesh we want to deform in front of the skeleton and tell it to bind. 3D distance calculations determine what bones are connected to which vertices and their weighting are based on distance to the bone. So I now have a binding skeleton and our mesh weighted.
In the deforming stage I take this information and calculate the inverse matrices for the binding skeleton and apply this to the mesh. This deforms the mesh to skeleton unity. I then multiply the unity mesh by the current user skeleton matrices for who ever is in front of the kinect. The idea is that the mesh should bind to the user skeleton and we should have a mesh that is deformable by the user.
However I cant seem to get it right. There appears to be stretching of the mesh and rotation instability as I move the joints (causing sharp jumps).
So I have three basic methods BindBones, GenerateInverseBindingSkeletonTransformedMesh and DeformMesh. The first determines the weighting of the mesh and what vertices are connected to which bones, the second transforms the mesh we have by the inverse of the binding skeleton matrices and the third takes that mesh and transforms it using the current user skeleton.
I've attached a screenshot exhibiting the behaviour, the binding and deforming skeleton are exactly the same thanks to the dummy but the mesh clearly isnt the same. With smooth skinning switched on it goes nuts, with smooth skinning off(only one bone deforming) it looks nearly right. Has anyone got any idea of what this behaviour might be caused by? I can provide more detail if required.