In my game I need enemies to turn towards the object they are chasing. So I tried to make a function to turn their rigid body towards the point their going to by applying a rotational impulse proportional to the dot product between the way they're facing and the direction they need to do to. Here is my code:

void lookAtPoint(btVector3 point, btRigidBody* body)
{
	btVector3 pos = body->getCenterOfMassPosition();

	btVector3 dir = (point - pos);
	dir.setY(0.0); //restrict to two dimentsions
	dir.normalize();

	btTransform t;

	body->getMotionState()->getWorldTransform(t);
	btVector3 forwards(t.getBasis()[2]);

	forwards.normalize();

	body->applyTorqueImpulse(btVector3(0.0f, forwards.dot(dir), 0.0f));
}

The problem is that the object seems to turn in the direction opposite to where it's supposed to even if I reverse the direction of the impulse. For example if the object is to their right they will turn left. Is their any problem with my approach or am I making a simple stupid mistake with my code?

The dot product is going to approach it's maximum value (1.0 in this case) as 'forward' and 'dir' approach the same direction. This means that your rotational speed increases as you orient towards the target. You could try,

body->applyTorqueImpulse(forwards.cross(dir))

although I am uncertain about the correct sign here.

-Josh

Taking the dot product between your current look vector and your desired look vector (this is what you are doing) doesn't tell you which direction (left or right) you need to get them to match. To figure out which direction you need to turn and by how much use the cross product instead. If you have two unit length vectors u and v then:

omega = u cross v

where the direction of omega specified the axis of rotation and the magnitude of omega is equal to the sine of the angle between u an v. You seem to be restricting yourself to 2 dimensions here so it's actual a bit easier. Here is some pseudo code....

float GetTopDownTurnDirection(const vector3& a_vCurrentLook,
                              const vector3& a_vDesiredLook)
{
    vector3 vOmega  = vector3::Cross(a_vCurrentLook, a_vDesiredLook);

    return vOmega.y
}