Is this an ML based system? I.e. does it have to be trained with a dataset of gestures ahead of time to recognize? Or can it compute raw pose data?

I'd call it raw, not trained so it will fit of the so called "unsupervised learning". Also, for the facial recognition I'm using the amazing Accord framework, which is built over the equally amazing Aforge but NOT for the gestures. I'm using my own code/logic for the gestures. Accord/Aforge seemed to slow for this kind of thing...or at least I didn't now how to setup for that. I have some screenshots too that demonstrate the way it operates...