I've implemented a vertex shader system to deform the model according to up to four vertex weights that is all working fine. You can assign a colour to each bone and the screenshot shows the model coloured based on its vertex weights. In animation mode, you can position the skeleton and set keyframes, and the model is deformed accordingly. You can then assign the keyframes to an animation slider and playback the animation and so on.
I originally had a GUI to assign the weights a little like Milkshape with a list of comboboxes with sliders, but I've actually just removed this as it was fiddly and not easy enough to use. I'm not quite sure what I'm going to implement instead. I've been watching YouTube videos of how things like Maya and 3DSMax handle rigging but not yet quite seen what I'm looking for.
Once I've implemented this though that pretty much covers the animation system. I can then think about texturing, which is the main missing component at the moment. After that it is really just a case of tidying up and adding a few features here and there but the bulk of the work is done.
I'd also like to improve the vertex shader so I don't have to use a full matrix palette as it seems quite limiting in terms of how many bones I can support. I've been reading up about just passing quaternion and translation vectors into the shader instead and generating the matrices on the fly but not sure this is worth it since this only halves the memory required by my calculations.
Seems a shame that at the moment I can only support about 60 bones. Sounds like a lot, but I can imagine hitting that limit quite quick on a fully rigged biped with fingers. Not that I intend to make such models, but when you're working on a general editor, you like to be as flexible as you can. I may have to bite the bullet and fall back on software for the animation but would prefer to avoid this if possible.
Got a few slowdowns as well that I need to address. Performance is generally quite good as has been the priority all the way through but seem to be hitting some issues since I implemented the vertex shader that supports matrix-palette-based animation so need to address that.
That's all. Thanks for reading.
The most intuitive way of assigning vertex weights I know of, is to first map them automatically as best as can based on a distance-to-bone based falloff and then allow to refine them using an airbrush tool.
Why is it exactly that you are limited to 60 bones?