DX11 GPU Skinning Problem

Recommended Posts

1 hour ago, B. / said:

which Format would be the best for GPU Skinning, maybe FBX?

Seems better, yes. But i'm totally no expert with file formats. There are many shipped games using collada - i would not drop this if you already invested some time. Maybe the missing data is stored elsewhere in the file? Maybe open a new collada related topic so people knowing better can help. (Say also which modelng app you use then.)

1 hour ago, B. / said:

And for CPU Skinning i would need a origin Vertex List and transform these and set it to the vertexbuffer for every fps?

Yes, but performance does not matter - just to proof you have correct data and your algorithm is right if you can't get it to work otherwise. (I have a debug visualization class that buffers lines, point or polygons with given color and drwas once per frame. This would do and is very useful for pretty much anything.)

Share on other sites

Hi Joel,

I found my mistake, the 12 vertex weights was linked with the 12 vertices pos array of the file and now i get the same result as assimp

So now i load the weigths and Bone Indices right and if i only send Matrix Identitys to the shader, the model will draw correct.

Send I the origin bone matrices, that i load from the file, I get a Science-Fiction effect by drawing the model. See image

So the last thing to do is now to send the right transformed matrices to the shader.

Here the code how i send these to the shader (The BindPose Matrix is invert and the result is a Matrix Idetity)

                List<Matrix> boneMatrices = new List<Matrix>();

foreach (Joint joint in this.bones)
{
Matrix m = MatrixHelper.CalculateMatrixFromParents(joint, Matrix.Identity);
}

return boneMatrices;

Anyone an idea how to solve the last stept?

Greets

Benjamin

Share on other sites

Try to transpose the matrix coming from file (row major vs. column major convention issue)

Try to reverse multiplication order of both matrices (unlikely)

Try combining both approaches.

Trial and error fun

Edited by JoeJ

Share on other sites

That only bring a new strange effect, see image

Heres again my code

Engine Code

        public static Matrix CalculateMatrixFromParents(Joint joint, Matrix world)
{
if (joint.Parent != null)
{
world = CalculateMatrixFromParents(joint.Parent, joint.Parent.Transform.Matrix) * world;
return world;
}
else
return joint.Transform.Matrix;
}

List<Matrix> boneMatrices = new List<Matrix>();

if (this.bones.Count != 0)
{

foreach (Joint joint in this.bones)
{
Matrix m = MatrixHelper.CalculateMatrixFromParents(joint, Matrix.Identity);
m.Transpose();
}
}

return boneMatrices;

float4 ApplyBoneTransform(Vertex input, float4 value)
{
if(HasBones)
{
float4x4 skinTransform = (float4x4)0;
skinTransform += BoneMatrices[input.BoneIndices.x] * input.Weights.x;
skinTransform += BoneMatrices[input.BoneIndices.y] * input.Weights.y;
skinTransform += BoneMatrices[input.BoneIndices.z] * input.Weights.z;
skinTransform += BoneMatrices[input.BoneIndices.w] * input.Weights.w;

float4 position = mul(value, skinTransform);

return position;
}
else
return value;
}

{
Pixel result = (Pixel) 0;

float4 posWorld = mul(ApplyBoneTransform(input, float4(input.Position.xyz, 1.0f)), World);
result.Position = mul(mul(posWorld, View), Projection);
result.Normal = normalize(mul(ApplyBoneTransform(input, float4(input.Normal.xyz, 1.0f)), WorldIT));
result.UV = input.UV;
result.View = ViewInverse[3] - mul(float4(input.Position.xyz, 1.0f), World);
result.Tangent = normalize(mul(ApplyBoneTransform(input, float4(input.Tangent.xyz, 1.0f)), WorldIT).xyz);
result.Binormal = normalize(cross(input.Normal, input.Tangent));

return result;
}

Greets

Benjamin

Share on other sites

Hmm... thinking of it, there might be still something wrong with the import or wiring the data properly.

What is really strange is that the mesh triangles seem to be teared apart. This indicates a logical error more likely than wrong math.

If only math would be wrong, the mesh would bend strangely, but it would not disconnect.

So i assume you have duplicated vertices due to different normals or UVs. You could look for those vertices (log their numbers or using debugger). If you find two vertices with the same position, they should have the same bone indices and weights. If not, import logic must be still wrong.

(After that: I see you draw white lines and circles, so you already have debug visuals. You could use them to do the whole skinning on CPU to find bugs more easily as suggested earlier.)

Share on other sites

Hi Joe,

you had totally right, after hours of checking my code, i found two fatal bugs i made.

The first one was a logic mistake by load the boneindices of the file and the second was to set the wrong offsets for weights and boneindices to the shader.

So i fix this and now the geometry will draw right and can be deform by the bone matrices, without disconnect the triangle shap

But there is still a last little bug. If i send the bone matrices in order of the code i post, i get a little scaling effect of the X Axis, so the mesh will get a little bit larger

Inverse order of multiplication the bone matrices has the same result. Transpose the value of multiplication of all matrices, the mesh will pressed like from a heavy hammer. Invert the result of multiplication, i get a scaling effect of the X Axis too, but it will be a little bit smaller.

Has anyone an Idea to fix this last step?

Greets

Benjamin

Share on other sites

If i import the Collada file back to Maya i get a warning message that the tranform of every single joint is not compatible with fbx, so its baked into trs?

Is not the Problem, that the gpu skinning should be only, if bone matrices animations exist, because my 3 bones has transform matrices who say where they should be in the world, but that is not the position of the vertices, so for a example, a bone with the x postion 3 and tranform with the binding vertices translate these 3 steps right on the x Axis and that is wrong right, because without a animation, every single vertex should be only tranform with a matrix ideity until the get animated?

So the transform matrix should be the difference of the old and new bone transform matrix around the postion of the bone?

Greets

Benjamin

Share on other sites
20 hours ago, JoeJ said:

(After that: I see you draw white lines and circles, so you already have debug visuals. You could use them to do the whole skinning on CPU to find bugs more easily as suggested earlier.)

Maybe it's time for this now. I'd start without any weights and linking each vertex to the closest bone. That's a bit easier but enough to understand verify any involved math.

EDIT:

The inverse bind matrix transforms the vertex from model space to local bone space

The animated bone matrix transforms from bone space to final world space

Edited by JoeJ

Share on other sites

As requested by PM, i'll try to give an example for better understanding...

I don't know how 'new' you are to this, but to me the key was to treat this all as a geometrical problem. Back then i was not aware this is an application of linear algebra, and i still think of this stuff in a pure geometrical way.

Let's start with an exercise of how to transform a vector from one space to another:

// our source space: (referring to the bone transform in rest position)
vec sx (1,0,0);
vec sy (0,1,0);
vec sz (0,0,1);
vec sp (2,3,4);
// and target space: (referring to the animated space)
vec tx (0,0,1);
vec ty (1,0,0);
vec tz (0,1,0);
vec tp (5,6,7);
// this is a skin vertex in global space (or model space, but usually that's the same if model and skelton are at the origin)
vec skin (3,3,3);

// first, we move the vertex from world space to the local source space
vec localInS = skin - sp; // position now relative to source origin, now care for orientation:
localInS = vec (
sx.Dot(localInS), // how far from source origin along the direction of its X axis?
sy.Dot(localInS), // how far from source origin along the direction of its Y axis?
sz.Dot(localInS)); // and Z
// vertex is now in source space. Key is here to understand how the dot product works.
// so we transformed the vertex from its modeled position to the bone of the skeleton that should affect in during animation.

// next, think of the target space as a animated variant of the skeleton bone.
// we already know the position relative to that bone, now all we need to do is to transform it back to world space but using animated transform
vec animatedSkin = tp + vec (
tx * localInS.x,
ty * localInS.y,
tz * localInS.z);

// that's it. We are done. If we have multiple bones affecting the vertex, we do the same calculation for each of them and add te a weighted final result, like:
vec result =
animatedSkin * 0.25 +
animatedSkin_2 * 0.25 +
animatedSkin_3 * 0.5;
// that's obvious, but just notice we lerp only resulting vectors istead the spaces, which is faster

If you understand this (take some time, imagine it geomatrically, visualize it to help your brain a lot...),

then you understood all math there is to know.

We could rewrite this code using matrices, the spaces already hint how as they have the same memory order than a 4x3 matrix, just adding a (useless) final row to get 4x4 it would look like this:

	float sMatrix[16] = {
1,0,0,0,
0,1,0,0,
0,0,1,0,
5,6,7,1};


You can then look your math library code to see it performs identical operations than dot products.

Thinking this further, e.g. transforming all 3 direction vectors of one matrix by another matrix, you already understood how 3D rotations work using matrices. Only requirement is to understand the dot product (!)

(This assumes OpenGL matirx order, DirectX transposed that convention because MS likes to have its own standards to force people into their great... you get me)

So in DirectX it would look like this:

	float sMatrix[16] = {
1,0,0,5,
0,1,0,6,
0,0,1,7,
0,0,0,1};


... which causes things like using simple multiplications instead dot product and vice versa - trial and error fun starts here.

I'll rewrite above code using matrices:

matrix mS = {...};
matrix mT = {...};
vec skin (3,3,3);

vec localInS = mS.Unrotate(skin - sp);
// or:
vec localInS = mS.Inversed().Transform(skin);
// or:
vec localInS = mS.InverseTransform(skin);

vec animatedSkin = mT.position + mT.Rotate(localInS);
// or:
vec animatedSkin = mT.Transform(localInS);

Try to follow this so all variants of the same thing makes sense.

Finally, let's combine both involved transforms to one (that's what i've missed in my very first post.)

	matrix combined = mS.Inversed() * mT; // or mT * mS.Inversed(), depending on the convention your math lib uses
vec animatedSkin = combined.Transform(skin);


Usually you can write just 'animatedSkin = combined * skin', but 'Transform' makes more sense eventually

So that's it. I hope this helps to understand the math, but i doubt it will help you much to fix your actual problem - this is expected.

There seems nothing wrong with how you do it, bug an be anywhere. I can only repeat my suggestions to do it all an CPU and visualize all steps... good luck!

Edit: To be sure here is how a space is defined if you're total noob

vec sx (1,0,0); // local orientation x axis

vec sy (0,1,0); // local orientation y axis

vec sz (0,0,1);

vec sp (2,3,4); // position

... same for matrices of course

Edited by JoeJ

Share on other sites

Hi Joe,

thank you very much again, to take the time to explian me it, but i see in your example, we assume that we animate the model, but whats with the case, i only load the model without animation matrices, so we dont have a target space, just to see the skinned model, do i need than to transform the vertices too, or only if i had animation matrices?

Greets

Benjamin

Create an account

Register a new account

• 9
• 13
• 41
• 15
• 14
• Similar Content

• By chiffre
Introduction:
In general my questions pertain to the differences between floating- and fixed-point data. Additionally I would like to understand when it can be advantageous to prefer fixed-point representation over floating-point representation in the context of vertex data and how the hardware deals with the different data-types. I believe I should be able to reduce the amount of data (bytes) necessary per vertex by choosing the most opportune representations for my vertex attributes. Thanks ahead of time if you, the reader, are considering the effort of reading this and helping me.
I found an old topic that shows this is possible in principal, but I am not sure I understand what the pitfalls are when using fixed-point representation and whether there are any hardware-based performance advantages/disadvantages.
(TLDR at bottom)
The Actual Post:
To my understanding HLSL/D3D11 offers not just the traditional floating point model in half-,single-, and double-precision, but also the fixed-point model in form of signed/unsigned normalized integers in 8-,10-,16-,24-, and 32-bit variants. Both models offer a finite sequence of "grid-points". The obvious difference between the two models is that the fixed-point model offers a constant spacing between values in the normalized range of [0,1] or [-1,1], while the floating point model allows for smaller "deltas" as you get closer to 0, and larger "deltas" the further you are away from 0.
To add some context, let me define a struct as an example:
struct VertexData { float[3] position; //3x32-bits float[2] texCoord; //2x32-bits float[3] normals; //3x32-bits } //Total of 32 bytes Every vertex gets a position, a coordinate on my texture, and a normal to do some light calculations. In this case we have 8x32=256bits per vertex. Since the texture coordinates lie in the interval [0,1] and the normal vector components are in the interval [-1,1] it would seem useful to use normalized representation as suggested in the topic linked at the top of the post. The texture coordinates might as well be represented in a fixed-point model, because it seems most useful to be able to sample the texture in a uniform manner, as the pixels don't get any "denser" as we get closer to 0. In other words the "delta" does not need to become any smaller as the texture coordinates approach (0,0). A similar argument can be made for the normal-vector, as a normal vector should be normalized anyway, and we want as many points as possible on the sphere around (0,0,0) with a radius of 1, and we don't care about precision around the origin. Even if we have large textures such as 4k by 4k (or the maximum allowed by D3D11, 16k by 16k) we only need as many grid-points on one axis, as there are pixels on one axis. An unsigned normalized 14 bit integer would be ideal, but because it is both unsupported and impractical, we will stick to an unsigned normalized 16 bit integer. The same type should take care of the normal vector coordinates, and might even be a bit overkill.
struct VertexData { float[3] position; //3x32-bits uint16_t[2] texCoord; //2x16bits uint16_t[3] normals; //3x16bits } //Total of 22 bytes Seems like a good start, and we might even be able to take it further, but before we pursue that path, here is my first question: can the GPU even work with the data in this format, or is all I have accomplished minimizing CPU-side RAM usage? Does the GPU have to convert the texture coordinates back to a floating-point model when I hand them over to the sampler in my pixel shader? I have looked up the data types for HLSL and I am not sure I even comprehend how to declare the vertex input type in HLSL. Would the following work?
struct VertexInputType { float3 pos; //this one is obvious unorm half2 tex; //half corresponds to a 16-bit float, so I assume this is wrong, but this the only 16-bit type I found on the linked MSDN site snorm half3 normal; //same as above } I assume this is possible somehow, as I have found input element formats such as: DXGI_FORMAT_R16G16B16A16_SNORM and DXGI_FORMAT_R16G16B16A16_UNORM (also available with a different number of components, as well as different component lengths). I might have to avoid 3-component vectors because there is no 3-component 16-bit input element format, but that is the least of my worries. The next question would be: what happens with my normals if I try to do lighting calculations with them in such a normalized-fixed-point format? Is there no issue as long as I take care not to mix floating- and fixed-point data? Or would that work as well? In general this gives rise to the question: how does the GPU handle fixed-point arithmetic? Is it the same as integer-arithmetic, and/or is it faster/slower than floating-point arithmetic?
Assuming that we still have a valid and useful VertexData format, how far could I take this while remaining on the sensible side of what could be called optimization? Theoretically I could use the an input element format such as DXGI_FORMAT_R10G10B10A2_UNORM to pack my normal coordinates into a 10-bit fixed-point format, and my verticies (in object space) might even be representable in a 16-bit unsigned normalized fixed-point format. That way I could end up with something like the following struct:
struct VertexData { uint16_t[3] pos; //3x16bits uint16_t[2] texCoord; //2x16bits uint32_t packedNormals; //10+10+10+2bits } //Total of 14 bytes Could I use a vertex structure like this without too much performance-loss on the GPU-side? If the GPU has to execute some sort of unpacking algorithm in the background I might as well let it be. In the end I have a functioning deferred renderer, but I would like to reduce the memory footprint of the huge amount of vertecies involved in rendering my landscape.
TLDR: I have a lot of vertices that I need to render and I want to reduce the RAM-usage without introducing crazy compression/decompression algorithms to the CPU or GPU. I am hoping to find a solution by involving fixed-point data-types, but I am not exactly sure how how that would work.
• By cozzie
Hi all,
I was wondering it it matters in which order you draw 2D and 3D items, looking at the BeginDraw/EndDraw calls on a D2D rendertarget.
The order in which you do the actual draw calls is clear, 3D first then 2D, means the 2D (DrawText in this case) is in front of the 3D scene.
The question is mainly about when to call the BeginDraw and EndDraw.
Note that I'm drawing D2D stuff through a DXGI surface linked to the 3D RT.
Option 1:
A - Begin frame, clear D3D RT
B - Draw 3D
C - BeginDraw D2D RT
D - Draw 2D
E - EndDraw D2D RT
F - Present
Option 2:
A - Begin frame, clear D3D RT + BeginDraw D2D RT
B - Draw 3D
C - Draw 2D
D - EndDraw D2D RT
E- Present
Would there be a difference (performance/issue?) in using option 2? (versus 1)
Any input is appreciated.

• Do you know any papers that cover custom data structures like lists or binary trees implemented in hlsl without CUDA that work perfectly fine no matter how many threads try to use them at any given time?
• By cozzie
Hi all,
Last week I noticed that when I run my test application(s) in Renderdoc, it crashes when it enable my code that uses D2D/DirectWrite. In Visual Studio no issues occur (debug or release), but when I run the same executable in Renderdoc, it crashes somehow (assert of D2D rendertarget or without any information). Before I spend hours on debugging/ figuring it out, does someone have experience with this symptom and/or know if Renderdoc has known issues with D2D? (if so, that would be bad news for debugging my application in the future );
I can also post some more information on what happens, code and which code commented out, eliminates the problems (when running in RenderDoc).
Any input is appreciated.

• Hi Guys,
I understand how to create input layouts etc... But I am wondering is it at all possible to derive an input layout from a shader and create the input layout directly from this? (Rather than manually specifying the input layout format?)