Your vectors aren't normalized. Normalize them first. That's required when using this equation.
Edit: (just to add an explanation for why the above matters)
To make it really clear, the dot product equation is:
A . B = ||A|| * ||B|| * cos(theta)
Where (A and B are your vectors, . is the dot product, and ||A|| means the magnitude of the vector A). In this form the equation, A and B are not required to be normalized (because the ||A|| * ||B|| part takes care of that... you'll see below).
Rearranging it, we see:
theta = acos((A . B) / (||A|| * ||B||))
Now, to normalize a vector A, you just do A / ||A||. I'll let An be the normlized version of A, where An = A / ||A||. Then you can plug it in as:
theta = acos(An . Bn)
Which is what you're trying to do, but your vectors aren't normalized, which is why you're getting problems.
So, to say it one more time:
theta = acos((A . B) / (||A|| * ||B||)) <-- General equation
theta = acos(An . Bn) <-- Special case of the generalized equation (simplified because for normalized vectors An and Bn, ||An|| * ||Bn|| = 1)
Edited by Cornstalks, 13 January 2013 - 11:08 PM.