To begin with, just change one matrix, eg rotate the shoulder of one arm 90 degrees. Is it parenting the rest of the arm, is it rotating correctly etc.
Secondly, I recommend having a debug avatar that is drawn just as spheres or sprites at the bone heads. There are no rotations applied. This will quickly pick up problems like:
Row vs column error
Z up vs y up error
Skeleton scaled, or rotated differently to skin.
With FBX, I've had problems where Blender exports the mesh and armature with different up's and you need to know which matrix to apply as you build the mesh to correct it. This is why I now only use collada, I have more visibility of what's going on.