My initial plan for skinning was to store all of the joint matrices for each model instance in one big texture. The advantage to this is that I can fit a lot more instances and bones into one draw call than I could if I was using constant registers.
The skinning system required me to create a texture manually for the first time, and I learned something rather interesting: You cannot directly access a texture which is in video memory. In order to put data into a VRAM texture, you need to create a second texture in system memory, fill it with the data, then update the texture in VRAM using IDirect3DDevice9::UpdateTexture(). Another interesting thing to note is that apparently a texture is a collection of surfaces. I originally thought it was just one big 3D array of data (for the different miplevels), but this does make more sense since each mipmap will be a different size.
Accessing the texture from the vertex shader was a bit more problematic than I was originally expecting. The first issue I encountered was that you cannot use tex*D() to access the texture (at least not in D3D9, it might work in D3D10+). The reason for this is that in a pixel shader, the miplevel is calculated based on the texture coordinate partial derivatives. Basically, if the texture coordinates changed a lot since the previous pixel, a lower miplevel is chosen in order to keep the pixel:texel ratio as close to 1:1 as possible. The derivatives can't be calculated in the vertex shader though, because we are working with vertices and not pixels. So to get around this, you need to ask for a specific miplevel by using tex*Dlod().
That should have been the end of the VTF (vertex texture fetch) problems, but then Present() started failing... I couldn't find anything wrong with the code, so I enabled debugging through the DirectX Control Panel to see if I it would tell me anything about the error. Apparently the driver was failing internally, but no further information was being given. I asked for help in the #gamedev IRC channel and we eventually figured out the problem; Apparently the Radeon X1000 series of graphics card, of which mine is a part of, does not support VTF. The really odd part of this is that one of the selling features of this series was its supposed support for SM3.0, which should have included support for VTF. ATI have a workaround for the problem, called "Render to Vertex Buffer", but at this point, using constant registers is looking better and better.
In the end, I decided to scrap the instancing code and use constant registers for skinning. At least the constant registers work like they should, so I can actually get 3D animation up and running.
I currently have single-joint skinning working, but the code for finding the keyframes to interpolate between is a real mess. I have one or two ideas on how to fix this, so we'll see how that goes tomorrow. It is kind of cool to see animation working, even if it is only a ball moving up and down.
[size="1"]Reposted from http://invisiblegdev.blogspot.com/