Sign in to follow this  

No body cares about me and my Skinning Shader/ Physics Engine problems!

This topic is 3284 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm having a really hard time figuring this out, I am trying to create a rag doll with PhysX. I ridged it right but I can't figure out how to get the matrices to work with my skinning shader. The problem is that I can only get a global world matrix for each bone. I thought I could inverse the offset matrix for the mesh's bones and then multiply it by the global pose of the actor for that bone to get the difference in rotation and position. It works for the position but the rotation is completely messed up and I have no idea how to fix it. If anyone has done rag dolls using any type of physics engine with a soft skinning shader please help me I'm completely lost and have been trying to fix this for days now. [depressed] What I'm doing now with the matrices:
	void Render( D3DXMATRIX mView, D3DXMATRIX mProj )
	{
		vector<D3DXMATRIX> matrices;
		matrices.resize(bones.size());
		D3DXMATRIX tmatrix;
		for(UINT i = 0; i < bones.size(); i++ )
		{
			bones[i]->getGlobalPose().getColumnMajor44( matrices[i] );
			D3DXMatrixInverse( &tmatrix, NULL, &mesh->bones[i].matrix );
			D3DXMatrixMultiply( &matrices[i], &tmatrix, &matrices[i] );
		}
		mesh->Render( matrices, mView, mProj );
		matrices.clear();
	}







Skinning Shader:
Texture2D txDiffuse
< 
string UIName = "Base Texture";
>;
SamplerState samLinear
{
    Filter = MIN_MAG_MIP_LINEAR;
    AddressU = Wrap;
    AddressV = Wrap;
};

TextureCube txEnvironment
<
string UIName = "Environment Map";
>;
SamplerState envSampler
{
	Filter = MIN_MAG_MIP_LINEAR;
	AddressU = Clamp;
	AddressV = Clamp;
	AddressW = Clamp;
};

float Environment_map_intensity
<
	string UIName = "Intensity";
	float UIMin = 0.000000;
	float UIMax = 1.000000;
	float UIStep = 0.01;
	string UIWidget = "slider";
> = 1.000000;

float4 Ambient  <
	string UIName = "Ambient";
> = float4( 0.47f, 0.47f, 0.47f, 1.0f );
	
float4 Diffuse  <
	string UIName = "Diffuse";
> = float4( 0.47f, 0.47f, 0.47f, 1.0f );

float4 lightPos : Position <  string UIName = "Light Position"; string Object = "PointLight";> ={0.0f, 0.0f, -10.0f, 0.0f};

float4x4 World				: WORLD;
float4x4 View				: WORLDVIEW;
float4x4 Projection			: PROJECTION;
float4x4 Bones[256]		: WORLDMATRIXARRAY;

struct VS_INPUT
{
	float4 pos			: POSITION;
	float3 norm			: NORMAL;
	float2 tex			: TEXCOORD0;
	float4 bw			: TEXCOORD1;
	float4 bi			: TEXCOORD2;
};

struct PS_INPUT
{
	float4 pos			: SV_POSITION;
	float4 pos2			: POSITION;
	float2 tex			: TEXCOORD0;
	float3 norm			: NORMAL;
};


PS_INPUT VS( VS_INPUT In )
{
	PS_INPUT Out = (PS_INPUT)0;
   	float4x4 skinTransform = 0;
    	skinTransform += Bones[In.bi.x] * In.bw.x;
    	skinTransform += Bones[In.bi.y] * In.bw.y;
    	skinTransform += Bones[In.bi.z] * In.bw.z;
    	skinTransform += Bones[In.bi.w] * In.bw.w;
    	Out.pos = mul(In.pos, skinTransform);
	Out.norm = mul(In.norm, (float3x3)skinTransform);
    	Out.pos = mul(mul(Out.pos, View), Projection);
	Out.tex = In.tex;
	return Out;
}

float4 PS( PS_INPUT input) : SV_Target
{
float4 color = txDiffuse.Sample( samLinear, input.tex );
float a = color.w;
color = (color * (dot(input.norm,normalize(input.pos2-lightPos)))) + ( Environment_map_intensity * txEnvironment.Sample( envSampler, -reflect( normalize( input.pos2 - View[3] ).xyz, input.norm.xyz )));
color.w = a;
return color;
}

technique10 Render
{
	pass P0
	{
		SetVertexShader( CompileShader( vs_4_0, VS() ) );
		SetGeometryShader( NULL );
		SetPixelShader( CompileShader( ps_4_0, PS() ) );
	}
}







[Edited by - CodaKiller on January 17, 2009 5:31:45 PM]

Share this post


Link to post
Share on other sites
I'm still having the problem, I thought I fixed it until I used a more complex rig. Has anyone done this before?!

Share this post


Link to post
Share on other sites
Hi,

Probably not of much help. But this is how I do my bone animation.

I calculate the local matrix by interpolating between two key frames (yes, I know this is mathematically not correct, but if the keyframes aren't too far apart appears ok). Then I multiply the local transform with anything from the parent object.

From the top of my head. I export everything from blender (custom exporter) and first multiply with the inverse of the rest pose (so every bone is moved completely 'down' once).

So first multiply with inverse rest pose then with all the transforms from the world and all the transforms in the hierarchy (ParentWorldTransform) to this one.

Note that the order A*B or B*A depends on the way stuff is exported so might be different in your situation.

The best way to debug is take a very simple situation, calculate by hand and verify in debug mode. Tedious but once the code is right you can probably forget about it :-)


void M3DAnimator::Transform( SVRefPtr<M3DEffect> Effect, M3DMatrix& ParentWorldTransform, UInt32 FrameNumber )

{

this->FrameNumber = FrameNumber;



M3DMatrix Local;

// Either a regular object or dropped here

if( Object )

{

Object->Frames.MatrixLerp( Local, this->FrameNumber );



D3DXMatrixMultiply( &WorldTransform->GetD3DXMATRIX(), &Local.GetD3DXMATRIX(), &ParentWorldTransform.GetD3DXMATRIX() );



Channel << Level4 << "Parent matrix" << ParentWorldTransform;

Channel << Level4 << "Local matrix" << Local;

Channel << Level4 << "WorldTransform" << WorldTransform;



for( std::vector<SVRefPtr<M3DAnimator> >::iterator ChildIter = Children.begin(); ChildIter != Children.end(); ++ChildIter )

{

(*ChildIter)->Transform( Effect, *WorldTransform, FrameNumber );

}



for( std::vector<SVRefPtr<M3DAnimator> >::iterator ChildIter = Hidden.begin(); ChildIter != Hidden.end(); ++ChildIter )

{

(*ChildIter)->Transform( Effect, *WorldTransform, FrameNumber );

}



}

}







In (one of) my vertex shader I then calculate how different bones( weighted) are pulling on the vertice


SPS VS_BO( SVS_BO i )
{
SPS Out;

Out.Pos =
mul( i.Vertex, InvMulPose0 ) * i.Weights.x +
mul( i.Vertex, InvMulPose1 ) * i.Weights.y +
mul( i.Vertex, InvMulPose2 ) * i.Weights.z +
mul( i.Vertex, InvMulPose3 ) * i.Weights.w;

float3 Normal =
mul(i.Normal, (float3x3) InvMulPose0 ) * i.Weights.x +
mul(i.Normal, (float3x3) InvMulPose1 ) * i.Weights.y +
mul(i.Normal, (float3x3) InvMulPose2 ) * i.Weights.z +
mul(i.Normal, (float3x3) InvMulPose3 ) * i.Weights.w;

Out.Pos = mul( Out.Pos, WorldView );
float3 Norm = mul( Normal, (float3x3)WorldView );

CalcSpot( Norm, Out.Pos, Out.Color, Out.Spec, NrOfSpotLights, true );

Out.Pos = mul( Out.Pos, Proj );

return Out;
}





Regards, Ron AF Greve


[/source]

Share this post


Link to post
Share on other sites
Quote:
Original post by Ron AF Greve
Hi,

Probably not of much help. But this is how I do my bone animation.

I calculate the local matrix by interpolating between two key frames (yes, I know this is mathematically not correct, but if the keyframes aren't too far apart appears ok). Then I multiply the local transform with anything from the parent object.

From the top of my head. I export everything from blender (custom exporter) and first multiply with the inverse of the rest pose (so every bone is moved completely 'down' once).

So first multiply with inverse rest pose then with all the transforms from the world and all the transforms in the hierarchy (ParentWorldTransform) to this one.

Note that the order A*B or B*A depends on the way stuff is exported so might be different in your situation.

The best way to debug is take a very simple situation, calculate by hand and verify in debug mode. Tedious but once the code is right you can probably forget about it :-)

*** Source Snippet Removed ***

In (one of) my vertex shader I then calculate how different bones( weighted) are pulling on the vertice

*** Source Snippet Removed ***

Regards, Ron AF Greve


[/source]


Are you talking about animation matrices or matrices from the physics engine?

Share this post


Link to post
Share on other sites
I'm not sure if I'm thinking about this right, I need to find the difference between two matrices to get a local matrix. I've been inverting one of them and multiplying it but the other.

I'm not sure if thats how I would do it?

Share this post


Link to post
Share on other sites
Hi,

Sorry mine was for regular animation I never used the physics engine. On the other hand is there any difference? I would assume that the physix engine just supply you with the final matrices, however since I never used it I might be wrong.

This is how I do it and if I look at your code most of it is in your code to assuming mesh->bones[i].matrix is the matrix of the bone in 'rest position' (but why isn't it precalculated since that shouldn't change?).

And where is the world transform or is this part all done in model space (then it would seem right)? The rotation would be off if it is not done in model space!

I think to make skinning work you have to have the bone back at the origin (so no world transform), because I figure that is where your vertices are. Now it always takes myself time and a lot of thinking to get it right. But I think you have to multiply the bone with the inverse of yours model world transformation (I'll take it that the pysics engine returns the world transformation?). So it is back in the neighbourhood of the models vertices (also not transformed into world space of course). But the bones stil do contain any transformation.

1. So take the inverse world transformation of you models and apply that to the bones returned from physix.

Now imagine that this is just the rest position. Obviously no transformation of the vertices should happen in this situation. However applying the bone transformation to them would move them i.e. we have to remove the 'rest position' transformation from the bones.

2. Multiply the matrix from the previous step with the inverse rest position of each bone.

Now what you have got is the remainimg transformation. This you can stage to the graphics card (check: for the rest position I would expect this to be the identity matrix).

After that the weighting and moving the vertices to world space would be done.

3. Apply weighting (i.e. the code in your VS is already there). and world/projection/view transformation.


Share this post


Link to post
Share on other sites
Quote:
Original post by Ron AF Greve
Hi,

Sorry mine was for regular animation I never used the physics engine. On the other hand is there any difference? I would assume that the physix engine just supply you with the final matrices, however since I never used it I might be wrong.

This is how I do it and if I look at your code most of it is in your code to assuming mesh->bones[i].matrix is the matrix of the bone in 'rest position' (but why isn't it precalculated since that shouldn't change?).

And where is the world transform or is this part all done in model space (then it would seem right)? The rotation would be off if it is not done in model space!

I think to make skinning work you have to have the bone back at the origin (so no world transform), because I figure that is where your vertices are. Now it always takes myself time and a lot of thinking to get it right. But I think you have to multiply the bone with the inverse of yours model world transformation (I'll take it that the pysics engine returns the world transformation?). So it is back in the neighbourhood of the models vertices (also not transformed into world space of course). But the bones stil do contain any transformation.

1. So take the inverse world transformation of you models and apply that to the bones returned from physix.

Now imagine that this is just the rest position. Obviously no transformation of the vertices should happen in this situation. However applying the bone transformation to them would move them i.e. we have to remove the 'rest position' transformation from the bones.

2. Multiply the matrix from the previous step with the inverse rest position of each bone.

Now what you have got is the remainimg transformation. This you can stage to the graphics card (check: for the rest position I would expect this to be the identity matrix).

After that the weighting and moving the vertices to world space would be done.

3. Apply weighting (i.e. the code in your VS is already there). and world/projection/view transformation.


Hmmm I get what your saying but even though I have a world matrix in the shader it's not actually being used, so I guess I'll have to do a trick to get the world matrix from the root bone which I believe I have an idea as to how I would do it. Alright I'll try that and tell you the results.

Share this post


Link to post
Share on other sites
Hi,

Thinking about it a bit more and looking in my code (it's been a while back since I programmed that part). This is how I do it

(*InverseRestBone)*(NewBonePositionModelSpace)=OneBoneTransform

ResultVerticeModelSpace = Vertice * OneBoneTransform

NewBonePositionModel is I think what you get back from the physix engine but multiply by inverse world transform if any.

The trick was that by having the InverseRestBone in the OneBoneTransform that when you later multiply the vertice with it you effectively move the vertex into local bone space at the origin then move it back to where it was in model space (with the additional transform) in one multiplication due to the NewBonePositionModelSpace.

If you (or I :-) ) got the matrices transposed you might have to reorder the multiplications.

Note: Sorry what I said about premultiplying didn't make sense (that only works in my case of fixed animations).

Share this post


Link to post
Share on other sites
Quote:
Original post by Ron AF Greve
Hi,

Thinking about it a bit more and looking in my code (it's been a while back since I programmed that part). This is how I do it

(*InverseRestBone)*(NewBonePositionModelSpace)=OneBoneTransform

ResultVerticeModelSpace = Vertice * OneBoneTransform

NewBonePositionModel is I think what you get back from the physix engine but multiply by inverse world transform if any.

The trick was that by having the InverseRestBone in the OneBoneTransform that when you later multiply the vertice with it you effectively move the vertex into local bone space at the origin then move it back to where it was in model space (with the additional transform) in one multiplication due to the NewBonePositionModelSpace.

If you (or I :-) ) got the matrices transposed you might have to reorder the multiplications.

Note: Sorry what I said about premultiplying didn't make sense (that only works in my case of fixed animations).


Isn't that what I was already doing? Also that other idea didn't work, the matrices are still messed up for some reason. Here is what I did:



void Render( D3DXMATRIX mView, D3DXMATRIX mProj )
{
D3DXMATRIX rb_world, inv_rb_rest, world, inv_world, tmatrix;
// get the world matrix from the root bone.
bones[mesh->root_bone]->getGlobalPose().getColumnMajor44( rb_world);
D3DXMatrixInverse( &inv_rb_rest, NULL, &mesh->bones[mesh->root_bone].matrix );
D3DXMatrixMultiply( &world, &inv_rb_rest, &rb_world );
D3DXMatrixInverse( &inv_world, NULL, &world );
vector<D3DXMATRIX> matrices;
matrices.resize(bones.size());
for(UINT i = 0; i < bones.size(); i++ )
{
bones[i]->getGlobalPose().getColumnMajor44( matrices[i] );
if( mesh->bones.size() > 0)
{
D3DXMatrixMultiply( &matrices[i], &matrices[i], &inv_world ); // step 1
D3DXMatrixInverse( &tmatrix, NULL, &mesh->bones[i].matrix );
D3DXMatrixMultiply( &matrices[i], &matrices[i], &tmatrix ); // step 2
}
}
mesh->Render( matrices, mView, mProj );
matrices.clear();
}


Share this post


Link to post
Share on other sites
Hi,

Yes, your first code does seem to be correct. Not sure about the last post. Its getting a bit late here and I have to get up early but this is what I think.


bones[mesh->root_bone]->getGlobalPose().getColumnMajor44( rb_world);
D3DXMatrixInverse( &inv_rb_world, NULL, rb_world); // Inverse world on root bone will take it back all the way but will also cancel its rest position
D3DXMatrixMultiply( &inv_world, &inv_rb_world, &mesh->bones[mesh->root_bone].matrix ); // Multiply by rest to get the root bone right again


vector<D3DXMATRIX> matrices;
matrices.resize(bones.size());
for(UINT i = 0; i < bones.size(); i++ )
{
bones[i]->getGlobalPose().getColumnMajor44( matrices[i] );
if( mesh->bones.size() > 0)
{
D3DXMatrixMultiply( &matrices[i], &matrices[i], &inv_world ); // bone back to model space
D3DXMatrixInverse( &tmatrix, NULL, &mesh->bones[i].matrix ); // Inverse rest position bone (to move vertice to the local bone space)
D3DXMatrixMultiply( &matrices[i], &tmatrix, &matrices[i]); // step 2 (I think this was right first but you swapped it?
}
}


Share this post


Link to post
Share on other sites
Quote:
Original post by Ron AF Greve
Hi,

Yes, your first code does seem to be correct. Not sure about the last post. Its getting a bit late here and I have to get up early but this is what I think.

*** Source Snippet Removed ***


It's having the exact same effect as my original code other then the model stays in the same position because I'm not using the world matrix in my shader, so basically this is another slightly more complex version of what I already had. [sad]

Share this post


Link to post
Share on other sites
I don't know if your vertex shader is correct.


PS_INPUT VS( VS_INPUT In )
{
PS_INPUT Out = (PS_INPUT)0;
float4x4 skinTransform = 0;
skinTransform += Bones[In.bi.x] * In.bw.x;
skinTransform += Bones[In.bi.y] * In.bw.y;
skinTransform += Bones[In.bi.z] * In.bw.z;
skinTransform += Bones[In.bi.w] * In.bw.w;
Out.pos = mul(In.pos, skinTransform);
Out.norm = mul(In.norm, (float3x3)skinTransform);
Out.pos = mul(mul(Out.pos, View), Projection);
Out.tex = In.tex;
return Out;
}




If I am not mistaken you blend the vertex output and not the matrix input. In other words you want something like this.


PS_INPUT VS( VS_INPUT In )
{
PS_INPUT Out = (PS_INPUT)0;
Out.pos = 0;
Out.pos += mul(In.pos, Bones[In.bi.x]) * In.bw.x;
Out.pos += mul(In.pos, Bones[In.bi.y]) * In.bw.y;
Out.pos += mul(In.pos, Bones[In.bi.z]) * In.bw.z;
Out.pos += mul(In.pos, Bones[In.bi.w]) * In.bw.w;

Out.norm = 0;

Out.norm += mul(In.norm, (float3x3)Bones[In.bi.x]) * In.bw.x;
Out.norm += mul(In.norm, (float3x3)Bones[In.bi.y]) * In.bw.y;
Out.norm += mul(In.norm, (float3x3)Bones[In.bi.z]) * In.bw.z;
Out.norm += mul(In.norm, (float3x3)Bones[In.bi.w]) * In.bw.w;

Out.pos = mul(mul(Out.pos, View), Projection);
Out.tex = In.tex;
return Out;
}




I am 100% sure on this one but with my understanding of transformation matricies
this part

float4x4 skinTransform = 0;
skinTransform += Bones[In.bi.x] * In.bw.x;
skinTransform += Bones[In.bi.y] * In.bw.y;
skinTransform += Bones[In.bi.z] * In.bw.z;
skinTransform += Bones[In.bi.w] * In.bw.w;



Is wrong

EDIT: Oh, it looks like that is already covered. My next piece of advice is to gain a better understanding of matrix and vertex transformations. As soon as you have that then you should be able to tackle these problems yourself.

Share this post


Link to post
Share on other sites
Quote:
Original post by HappyCoder
I don't know if your vertex shader is correct.

*** Source Snippet Removed ***

If I am not mistaken you blend the vertex output and not the matrix input. In other words you want something like this.

*** Source Snippet Removed ***

I am 100% sure on this one but with my understanding of transformation matricies
this part
*** Source Snippet Removed ***
Is wrong

EDIT: Oh, it looks like that is already covered. My next piece of advice is to gain a better understanding of matrix and vertex transformations. As soon as you have that then you should be able to tackle these problems yourself.


Your shader has the exact same effect but it's more complex.

Share this post


Link to post
Share on other sites
Here is before the object hits the ground or before rotations happen and after rotations happen. Both Ron AF Greve's and HappyCoder's ideas have the same effect as my original code. The bottom pictures are whats happening in the PhysX Visual Debugger and the objects in my engine should move in a similar fashion but clearly they are no...



[Edited by - CodaKiller on January 19, 2009 3:12:40 AM]

Share this post


Link to post
Share on other sites
I know whats wrong but no matter what I've tried I can't get it to work right, I need to change the center of rotation on the matrices to line up with the position of the bones.

Share this post


Link to post
Share on other sites
Ahhhhh!!! I'm going crazy, I can't get it to work! Has anyone used a physics engine with a skinning shader?! Please say something if you have!

Share this post


Link to post
Share on other sites
Sign in to follow this