If I set the physics after AI, physics will dominate the final world transformations, which totally erased the world transformations calculated from the AI, or vice versa.
I want to calculate the relative transformation when physics is in effect, how do I do that with btMotionState or derived classes with getWorldTransform overrided?
Thanks
Jack

Why is the AI concerned with world transformations? I would think the AI would determine things like picking actions, or what animation to play next...

If you override transforms after physics you run into the problem that you might break the physics constraints or move into penetration. You can take control over a physics object using ghost objects. A ghost object takes part in the broadphase, but not in the physics simulation. You are responsible for not moving the ghost objects into penetration (e.g. using raycast or volume sweeps).

I assume you are directly modifying position of objects?

If that is the case, you should make the AI set applied force or velocity, for example. Not the position itself. It's a common mistake with many physics engine with broad-phase (engines that will solve the position and angle of objects, the reactions).

I don't know how to figure out the forces applied to the object affected....

void AgentMotionState::getWorldTransform(btTransform& worldTrans) const 
{
btTransform t;
btDefaultMotionState::getWorldTransform(t);  
D3DXMATRIX l_mat;
D3DXMatrixIdentity(&l_mat);
if (m_object)
{
	// DX TRANSFORM
	l_mat = BT2DX_MATRIX(t);
	D3DXVECTOR3 scale, pos;
	D3DXQUATERNION quat; 
	D3DXMatrixDecompose(&scale, &quat, &pos, &l_mat);
	Transform* l_transform = m_object->FindComponentByType();
	// if setting positions here, the physics will dominate over AI 
	l_transform->setPosition2(pos);
	l_transform->setRotation(quat);
}
// BULLET TRANSFORM
worldTrans = t;
}

You need to figure out which part of the object is responsible for what actions.  Right now you have physics and AI both trying to move an object around, and that just wont work... clearly.    So, let the physics be responsible for the object's transform and let the AI determine what it wants to do, and then tell the physics system what it wants through setting forces or impulses.

9 minutes ago, lucky6969b said:

I don't know how to figure out the forces applied to the object affected....
 

Start with F = ma and work from there.  You should know what mass your objects have, and what speed you want, so just accelerate them until they reach that speed and then set accel back to 0.  Or, if your objects dont need to accelerate, just give them impulses.

Okay, When updating AI after the physics, somehow the AI can "overpower" what the physics is trying to do...
and the AI is futile, the physics will take over.
Thanks
Jack

But if I let the physics run in its own thread, this has some advantages when I don't update the physics too much by leveraging it some time to sleep,the AI actually has time to counteract with what the physics is currently doing. Because physics is continuous, if I divide the 1.0 with the frame rate, I can get some continuous feel without the physics hogging the world transformations all the time, when the physics thread is sleeping, the AI can do its work now...
Thanks
Jack

3 hours ago, lucky6969b said:

But if I let the physics run in its own thread, this has some advantages when I don't update the physics too much by leveraging it some time to sleep,the AI actually has time to counteract with what the physics is currently doing. Because physics is continuous, if I divide the 1.0 with the frame rate, I can get some continuous feel without the physics hogging the world transformations all the time, when the physics thread is sleeping, the AI can do its work now...
Thanks
Jack

I'm not sure I understand, do you still have both the AI and physics setting your object's transforms?

I meant when the physics thread goes to sleep, the AI can take control of the world transformation.