Jump to content

  • Log In with Google      Sign In   
  • Create Account


[DX9] Implementation of Meshcontainers and Frames


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
16 replies to this topic

#1 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 22 November 2011 - 01:33 AM

Hello

I have read the Frank Luna paper on "Skinned Mesh Character Animation with Direct3D 9.0c", some tutorials and a few topics on this forum. But I am still confused about the relationship between meshcontainers and frames(bones). I know the "theory" but don't understand exactly how it is implemented in DX9.

It seems that the vertex format is not loaded as a skinned mesh. Is this specific for X. files?

A model mesh or skin is copied in to a meshcontainer, and the meshcontainer is pointed to by a frame. Does that mean that you can not have more meshcontainers then bones? Does it matter which bone is pointing to which meshcontainer? Or is it just a way of "linking" them? If I can have a model split up in several meshcontainers, when do I want to do this? Will this "disconnect/break up" and not make a continuous skin/mesh? For an RPG game, would you have the "body armor piece" connected to the body bone etc?

Why is the implementation this way, why not just load everything from file directly in to an object that has everything ready for drawing?

Since you have an palette of transform matrices, each vertex has indices pointing to the ones that affect it and a weight to each one, right?

Is there some visual representation of how these things fit together? It is hard for me to grasp where things are during the loading process and during run-time.

Any comment appreciated.

Sponsor:

#2 BornToCode   Members   -  Reputation: 923

Like
0Likes
Like

Posted 22 November 2011 - 03:32 PM

Hello

I have read the Frank Luna paper on "Skinned Mesh Character Animation with Direct3D 9.0c", some tutorials and a few topics on this forum. But I am still confused about the relationship between meshcontainers and frames(bones). I know the "theory" but don't understand exactly how it is implemented in DX9.

It seems that the vertex format is not loaded as a skinned mesh. Is this specific for X. files?

A model mesh or skin is copied in to a meshcontainer, and the meshcontainer is pointed to by a frame. Does that mean that you can not have more meshcontainers then bones? Does it matter which bone is pointing to which meshcontainer? Or is it just a way of "linking" them? If I can have a model split up in several meshcontainers, when do I want to do this? Will this "disconnect/break up" and not make a continuous skin/mesh? For an RPG game, would you have the "body armor piece" connected to the body bone etc?

Why is the implementation this way, why not just load everything from file directly in to an object that has everything ready for drawing?

Since you have an palette of transform matrices, each vertex has indices pointing to the ones that affect it and a weight to each one, right?

Is there some visual representation of how these things fit together? It is hard for me to grasp where things are during the loading process and during run-time.

Any comment appreciated.


The best way to think about this is that the way skinned mesh works is that. You have n Bones where each Bone contains Position/Rotation or any additional info you need. Then you have Frames where each Frame contains information for each Bones. So if you had 10 bones in the model and you have to 2 frames then you would just take the number of bones*number of frames to figure out how long the animation will be. Then you have the MeshContainer which contains an handle to the Frames. One of the problems with D3DXFRAME is that it store Bone Transformation which is a really bad design because it makes it almost impossible to have two of the same models with the same vertexbuffer using an different animation. Here is an example of how i have everything set up in my engine.

struct Bone
{
GLGXVECTOR3 position;
GLGXQUATERNION rotation;
};

struct Frame
{
  Bone* bones;
}
class MeshContainer
{
  Frame* frames;
};

The vertices holds the vertex weight and bone index which they are been influence by.

#3 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 23 November 2011 - 08:31 AM

Frames as in bones or frames as in render a frame?

So if you had 10 bones in the model and you have to 2 frames then you would just take the number of bones*number of frames to figure out how long the animation will be.


What? animation length depends on what timeline the animation keys are at and how fast I am playing the track.

I know the theory, but confused about the DX9 implementation. Do most games use the DX animationController or do they have a custom made one?

A frame struct is the bone in my understanding.

Can you MSPAINT me a quick overview of how frames, meshcontainers, animcontroller etc fit together during loading and run-time?

Thanks for the comments, keep em coming :)

#4 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 25 November 2011 - 08:04 AM

bump for great justice :)

Questions in my first post still unanswered

#5 Steve_Segreto   Crossbones+   -  Reputation: 1519

Like
1Likes
Like

Posted 26 November 2011 - 07:40 AM

Let me take a crack at explaining this to you, though I think Frank Luna's paper says it clearer ( I know you've read it already ).

Question 1:

It seems that the vertex format is not loaded as a skinned mesh. Is this specific for X. files?

Answer 1: I assume when you say vertex format you mean vertex declaration and when you say it is not loaded as a "skinned mesh" you mean the vertex declaration does not have blend weights and blend indices in it after you load an X file using one of the D3DX APIs. I can't answer if this is specific only to .X files because there are tons of different file formats out there. Let's just assume the answer is "yes", if you load an X file using D3DXLoadMeshFromHierarchy() then since you are required to provide the overloaded implementation of the loading classes, the vertex format is not correctly expanded to include "skinning data" and you must do it yourself as shown in Luna's examples.

Question 2:

A model mesh or skin is copied in to a meshcontainer, and the meshcontainer is pointed to by a frame. Does that mean that you can not have more meshcontainers then bones?

Answer 2:

You can have more meshcontainers than bones. There is a field in the MeshContainer structure called "Next" which allows you to form an arbitarily long linked list of MeshContainers underneath any bone (frame) that you wish.

Question 3:

Does it matter which bone is pointing to which meshcontainer? Or is it just a way of "linking" them?

Answer 3:

The skinned mesh is represented as a frame hierarchy. In a hierarchy, being above, below or at the same level matters. In a frame hierarchy, if you attach a meshcontainer underneath a particular frame, the meshcontainer's vertex positions will be transformed by all the frame transformations above it (all the way up to the root). This is how weapons and shields are attached to skinned characters in games. The meshcontainer for a weapon is linked underneath the "hand" bone. Because of that the weapon is drawn at the current position of the "hand" bone each frame.

Question 4:

If I can have a model split up in several meshcontainers, when do I want to do this? Will this "disconnect/break up" and not make a continuous skin/mesh? For an RPG game, would you have the "body armor piece" connected to the body bone etc?

Answer 4:

You can have a model split up into several meshcontainers, real games do this all the time for player body part customization. The meshcontainers won't be continuous, but if your artists model them so they overlap a bit, nobody will notice. I think you are confusing the term "connected" with two different things here. The first term I like to call "attached", like when you link a meshcontainer underneath a bone in the hierarchy. A weapon meshcontainer could be attached to a hand bone, this does not mean that the weapon vertices are influenced by the hand bone during the vertex skinning however. It just uniformly transforms *ALL* the weapon vertices by the "toRoot" transform of the hand bone. So the second term is called "influenced". When a meshcontainer is influenced by a bone it means the bone appears in the blend indices list for one or more vertices of this meshcontainer's vertex buffer. It means the final world space position of that bone will deform one or more of the vertices of this meshcontainer. In your question above, the "body armor piece" should be influenced by the bones that make up the body (like the spine, pelvis, hips, clavicles, etc). The body armor piece would not be attached to those bones, it would be attached under the "Scene Root" bone most likely.

Question 5:

Why is the implementation this way, why not just load everything from file directly in to an object that has everything ready for drawing?

Answer 5:

I assume you mean the D3DX meshcontainer/frame and .X file terminology? The answer is really anybody's guess. As I said, most people don't use the D3DX allocation hierarchy or the ID3DXAnimationController in real products and most real engines can load everything into a scene graph that is ready for rendering. You still need to load your data hierarchically or else forward kinematics doesn't work.

Question 6:

Since you have an palette of transform matrices, each vertex has indices pointing to the ones that affect it and a weight to each one, right?

Answer 6:

There are two sides to the matrix palette in both Luna's and the DX9 examples. On the CPU side the matrix palette is just an array of pointers to the TransformationMatrix from particular frames of the skinned mesh hierarchy. In this way when the CPU hierarchically updates the TransformationMatrix field of each frame in the recursive step, the matrix palette is automatically up-to-date. Each frame the matrix palette is uploaded to the GPU. On the GPU side, the vertices coming into the vertex shader do have up to 4 integer indices which point into the matrix palette as it sits in the constants registers of the GPU. As you stated each vertex also has an array of up to 4 weights (which must sum to 1.0f (100%)). Inside the vertex shader the incoming local vertex position is adjusted via a weighted averaging (as shown in Luna's paper). After this the vertex position (and normal) can be transformed by world, view, projection matrices to enter clip space.

Question 7:

Is there some visual representation of how these things fit together? It is hard for me to grasp where things are during the loading process and during run-time.

Answer 7:

I don't have time to draw such a diagram, but Luna actually has included a better diagram than his online paper has in his book, "Intro to 3d game programming in dx9 - a shader approach".

#6 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 28 November 2011 - 05:03 PM

Thx for those awnsers.

One thing I still don't undertand; after D3DLoadMeshHierarcyFromX() is called I am supposed to "link frames" to their mesh containers, i.e populate the matrix palette with pointers. However it seems to me that after D3DLoadMeshHierarcyFromX() the frames are only initialized and not populated with any mesh container pointers, i.e D3DXFRAME.pMeshContainer is still blank. How can then the tutorial check pFrame->pMeshContainer during the linking step? What am I missing?

Again, thx.

#7 Steve_Segreto   Crossbones+   -  Reputation: 1519

Like
1Likes
Like

Posted 28 November 2011 - 05:51 PM

Thx for those awnsers.

One thing I still don't undertand; after D3DLoadMeshHierarcyFromX() is called I am supposed to "link frames" to their mesh containers, i.e populate the matrix palette with pointers. However it seems to me that after D3DLoadMeshHierarcyFromX() the frames are only initialized and not populated with any mesh container pointers, i.e D3DXFRAME.pMeshContainer is still blank. How can then the tutorial check pFrame->pMeshContainer during the linking step? What am I missing?

Again, thx.


Hi Tispe,


Can you please post the code snippet that is confusing you? You have two creation methods you must implement for an ID3DXAllocateHierarchy, CreateFrame and CreateMeshContainer. D3DX will link mesh containers to frames after the D3DXLoadMeshHierarchyFromX() API completes provided you returned non NULL frames and mesh containers at each of these creation methods.

#8 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 30 November 2011 - 07:08 AM

Hi again

I didn't think about D3DXLoadMeshHierarchyFromX() beeing able to initialize pFrame->pMeshContainer, sibling and child on its own after the allocation functions has been run. I guess that is whats going on.

So "attached" MeshContainers are recognized via the D3DXFRAME:pMeshContainer, and "influenced" MeshContainers are recognized via the matrix pallete.

If I ATTACH my gun to the hand bone, it gets translated to where the hand is, and if Boots are attached to the feet they get translated to where the feet bones are.

Lets say a game where the character can switch weapon, hat, jacket, gloves, pants and boots they all get attached to their respective bones. But the glove meshcontainer is only influenced by fingers, hand and forearm bones.

MS Tiny.x has just one mesh, meaning one meshcontainer. Probably linked to the root frame. And that MeshContainer is influanced by all bones.

It is kinda hard to grasp without an illustration. Lets say I want earmuffs and a hat, I would need two meshcontainers for each both attached to the head, meaning I have to chain them D3DXMESHCONTAINER* pNextMeshContainer.

Posted Image

#9 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 30 November 2011 - 03:08 PM

OK, my second attempt to figure out how this all fit together.

I know atleast how the hand vertices are blended (by using up to four matrices of the pallete).

What I don't quite get is how a hand mesh is translated to the hand bone. Obviously I need to multiply all the hand vertices by the hand CombTransformationMatrix, but the four bone indices of a vertex is supposed to add up to 1.0. Do I need to use the first of the four bone indices and do a 100% multiplication to translate the hand, then use the remaining three indices and multiply by up to three bones that also add up to 100%? This would mean a 200% multiplication. I figure the Offset Matrix is for another purpose, to translate from bone space to model space.

Please enlighten me about when the hand vertices are first transformed to the correct frame position.


As demonstrated in the below picture, when does the hat acutally get placed ontop of the head. Do I use the first BoneIndex for a 100% multiplication?

Posted Image

#10 Steve_Segreto   Crossbones+   -  Reputation: 1519

Like
1Likes
Like

Posted 30 November 2011 - 11:55 PM

OK, my second attempt to figure out how this all fit together.

I know atleast how the hand vertices are blended (by using up to four matrices of the pallete).

What I don't quite get is how a hand mesh is translated to the hand bone. Obviously I need to multiply all the hand vertices by the hand CombTransformationMatrix, but the four bone indices of a vertex is supposed to add up to 1.0. Do I need to use the first of the four bone indices and do a 100% multiplication to translate the hand, then use the remaining three indices and multiply by up to three bones that also add up to 100%? This would mean a 200% multiplication. I figure the Offset Matrix is for another purpose, to translate from bone space to model space.

Please enlighten me about when the hand vertices are first transformed to the correct frame position.


As demonstrated in the below picture, when does the hat acutally get placed ontop of the head. Do I use the first BoneIndex for a 100% multiplication?

Posted Image



This really depends on how your code traverses the frame hierarchy during the render loop. If it traverses it from top (Head) to bottom, then the Hat will be drawn first. You do not however show the contents of the meshcontainer's vertex buffer which is where the bone indices and weights are stored (1 per vertex). So I cannot guess how the blending happens.


I definitely think you should re-read Luna's paper about blending you are confused by how the weighted averaging of a vertex position occurs. First you transform all the frame matrices in your hierarchy using the current time and a set of keyframes. Then you render each meshcontainer as you encounter it using your traversal code in your render loop (usually top-down recursive). For each vertex of a meshcontainer, you take the 4x4 bone transform matrix referenced by that vertices bone index and transform the position by it and multiply by the bone weight. Then you add that to the next weighted position. Do this up to four times and you have the final Object space position of that vertex. Now multiply it by World, View, Projection matrix and its in clip space.

#11 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 01 December 2011 - 02:12 AM

hmmm.

Lets say a hand-mesh in centered in hand-space at the origin xyz=(0, 0, 0) and the hand-bone is at xyz=(50,100,20) in model-space. Each vertex is then multiplied by the Offset Matrix to align the hand-mesh to maybe xyz=(5,6,8), but it is not transformed to any specefic bone location yet. Now, why are there no intermediate transform between doing the Offset transform and vertex blending? I would think that if we went straight away to vertex blending after Offset the hand would only move its fingers at location xyz=(5,6,8). Or does the transform to xyz=(50,100,20) occur simultaneously when applying the four weighted matrices to deform the hand-mesh?

#12 Steve_Segreto   Crossbones+   -  Reputation: 1519

Like
1Likes
Like

Posted 01 December 2011 - 06:47 AM

hmmm.

Lets say a hand-mesh in centered in hand-space at the origin xyz=(0, 0, 0) and the hand-bone is at xyz=(50,100,20) in model-space. Each vertex is then multiplied by the Offset Matrix to align the hand-mesh to maybe xyz=(5,6,8), but it is not transformed to any specefic bone location yet. Now, why are there no intermediate transform between doing the Offset transform and vertex blending? I would think that if we went straight away to vertex blending after Offset the hand would only move its fingers at location xyz=(5,6,8). Or does the transform to xyz=(50,100,20) occur simultaneously when applying the four weighted matrices to deform the hand-mesh?



The time to consider the bone offset matrix is when your C++ program is setting up the "workingPalette" before sending it into the GPU as a 4x4 matrix array for the vertex shader to use during the skinning step.

The MultiAnimation sample calls an array of transform matrices used by a particular skin partition a "workingPalette". D3DX calls skin partitions ID3DXSkinInfo. A skin partition contains the minimum set of bones that are accessed by any vertex in this skin's mesh during the skinning step.

Typically if you want to have a very customisable player model, you define several different skin partitions (head, arms, body, legs, boots, etc) and attach them all to the root bone. The working palette for each skin partition gives a list of the bones that influence the vertices of this partition.

Right before you render a skin partition, you have a working palette of 4x4 matrices numbered from 0 .. W-1 and a complete frame hierarchy of bone matrices which you can flatten into an array of 0 .. B-1 bones. Note that W need not equal B but W <= B. Each vertex of the skin has a bone index in the 0 .. W-1 space (with a corresponding weight).

Now you fill in each matrix of the working palette by multiplying the bone offset transform[ 0 .. W-1 ] with the bone transform[ 0 .. W-1 ]. You must first map the range 0 .. W-1 back to 0 .. B-1 because the bone offset transform and the bone transforms are actually stored in the complete list of skeleton bones, the working palette for a skin partition just has the product of them.

Now upload the working palette to the shader and you can access the working palette using the index found in each vertices array of blend indices. Let me again state the blend index from a vertex is not necessarily the bone index in the master skeleton but the working palette index.

For a simple skinned mesh you might just have one single skin partition and would therefore have a 1:1 mapping between the working palette and the master skeleton list.

For a more complicated character, let's imagine he has an astounding 120 bones in his main skeleton. He has 5 skin partitions (head, body, arms, legs, gloves, boots). The head skin partition has a working palette with 10 of the 120 bones. The body skin partition has a working palette with 35 of the 120 bones (maybe some of the same as the head even), etc, etc. The sum of the sizes of the working palettes can be larger than the total number of bones in the character.

Breaking a character up into skin partitions like this helps because there is a limit on how many constants registers you can upload to a vertex shader and each matrix takes 4 float4 vectors (you can be tricky and store a transform as a quaternion and a translation and get down to half the size). So to draw any piece of the character above, the most you would have to upload is 35 4x4 matrices rather than 120 (assuming the body skin partition was the largest).

#13 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 02 December 2011 - 02:28 AM

Typically if you want to have a very customisable player model, you define several different skin partitions (head, arms, body, legs, boots, etc) and attach them all to the root bone.


Thank you for reading but I don't think you awnsered my question. What I am confused about is this:
Given a "boots" mesh container, this mesh is passed unaltered to the shader and each vertex is only transformed ONCE by the workingPalette right (4 bone weights etc)?

When you say "attach them all to the root bone", does this mean the root Frame D3DXFRAME:pMeshContainer and its D3DXFRAME:pNextMeshContainer are chained for all these partitions, all other frames have D3DXFRAME:pMeshContainer = NULL?

I don't see in the code how a partition is attached mathematically to any frame.

What I think happens:
Each mesh partition is seperate in their own meshcontainers. And each partition is in ther own space centered at the origin xyz=(0, 0, 0).
If all meshcontainers where to be drawn without beeing transformed, they would all appear ontop of eachother at the origin.
Each partition is therefore moved to their correct position, this is where the "attached" to a frame comes to my mind. If a meshcontainer is attached to a frame using D3DXFRAME:pMeshContainer then HOW is that meshcontainer transformed to the correct frame in model space?

What determines if the "arm mesh" is positioned at the arm bone? All I see is the workingPallete influencing the "arm mesh" vertices locally to deform the skin, not move it around in model space.

If you could try to explain in the following order:
1. What position is a vertex at when it is loaded from file
2. What position has a vertex as it sits in the MeshContainer
3. Is the Vertex position still the same when the MeshContainer is attached to a frame (loading process)
4. Is vertex still around the origin (as the mesh is modeled in its own space) when it is passed to the vertex shader?
5. Does the four influencing weighted combinedTransformMatrices*OffsetMatrices moving the Vertex from the "origin" position to its correct position on the model?

#14 Steve_Segreto   Crossbones+   -  Reputation: 1519

Like
1Likes
Like

Posted 02 December 2011 - 07:54 AM

A partition would be attached mathematically to a frame if you render the meshcontainer while recursing the frames.

1. Usually specified relative to (0,0,0)
2. Same as in file
3. Yes still the same vertex position
4. Yes
5. Yes

Since you like DirectX '.X' files have a look here: ModelViewer.zip

It's a zipped file (50 MB dowload) that has a library of X files and textures converted from the characters and monsters of WildTangent's FATE game. I do not own the rights to the artwork or models, but I have converted the information from the MDL/SMA/SMS file formats natively used by the game to the X file format.

Included is a small model viewer app - the binary for it is in bin\ModelViewer.exe. If you associate files of type 'X' with the application than you can double-click to examine each character and monster file in the library under "bin\FateCombined"

The controls for the model viewer are:

'1' - Cycle to next animation
'W' - Wireframe
'S' - Solid
'V' - Show bones
'X' - Toggle skin drawing on/off
'SPACE' Single step through keyframes
'ENTER' Stop single step

Additionally you can open any of the X fies using notepad.exe and view them as text files. They are well formatted with regard to whitespace (unlike Tiny.x) and examining the contents of them should give you a better idea of how skinned meshes work.

#15 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 02 December 2011 - 08:12 AM

Thanks

When you say "attach them all to the root bone", does this mean the root Frame D3DXFRAME:pMeshContainer and its D3DXFRAME:pNextMeshContainer are chained for all these partitions, all other frames have D3DXFRAME:pMeshContainer = NULL?

Also: is the MDL/SMA/SMS file formats open? If not what mothod did you use to figure out file layout?

#16 Steve_Segreto   Crossbones+   -  Reputation: 1519

Like
1Likes
Like

Posted 02 December 2011 - 09:46 AM

Thanks

When you say "attach them all to the root bone", does this mean the root Frame D3DXFRAME:pMeshContainer and its D3DXFRAME:pNextMeshContainer are chained for all these partitions, all other frames have D3DXFRAME:pMeshContainer = NULL?

Also: is the MDL/SMA/SMS file formats open? If not what mothod did you use to figure out file layout?



Yeah that's usually how people do it, a linked list of skin partitions all attached under the root bone. But your mileage might vary.

I used a hex editor and Luna's paper to figure out the formats. There are lots of identity matrices in frame hierarchies so once you know what an IEEE float looks like in hex, you can easily identify the identity matrix in the data file. From that point on you just play sudoko with the file until you know what all the bytes mean.

You can do it to any file, like I did it for Mortal Kombat SSF files, Zelda BMD files, DAOC/Oblivion NIF files, etc. Somebody had already published the format of the NIF file, so I didn't have to do too much for that one.

And the funny thing was after examining all those games I realized that they were all pretty much doing skinned meshes the same. They used skinning to get meshes to deform as the skeleton animated and they used attachment to get a mesh to follow one particular bone (usually rigidly, but some of the games attached one skinned mesh to the bone of another one - like an animated character holding an animated whip in their hand and cracking it).

Examining the large number of assets these games have also convinced me of the drawbacks of the D3DX animation system, though it is sufficient for example and learning purposes, which is likely what it was intended for.

#17 Tispe   Members   -  Reputation: 1031

Like
0Likes
Like

Posted 04 December 2011 - 11:55 AM

Thanks for that information.

What paper are you refering to?

Once I figure out the file format for the game I wish to borrow assets from, it is just a matter of writing my own D3DXLoadMeshHierarchyFromX() function specefic for that file format. On that topic, what else behind the scenes like populating the pFirstSibling, pFirstChild, pMeshContainer and adding animation sets to the AnimationController would I need to do in such a function?




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS