Your vectors aren't normalized. Normalize them first. That's required when using this equation.

**Edit**: (just to add an explanation for why the above matters)

To make it really clear, the dot product equation is:

**A** . **B** = ||**A**|| * ||**B**|| * cos(theta)

Where (**A** and **B** are your vectors, . is the dot product, and ||**A**|| means the magnitude of the vector **A**). In this form the equation, **A** and **B** are not required to be normalized (because the ||**A**|| * ||**B**|| part takes care of that... you'll see below).

Rearranging it, we see:

theta = acos((**A** . **B**) / (||**A**|| * ||**B**||))

Now, to normalize a vector **A**, you just do **A** / ||**A**||. I'll let **A**_{n} be the normlized version of **A**, where **A**_{n} = **A** / ||**A**||. Then you can plug it in as:

theta = acos(**A**_{n} . **B**_{n})

Which is what you're trying to do, but your vectors aren't normalized, which is why you're getting problems.

So, to say it one more time:

theta = acos((**A** . **B**) / (||**A**|| * ||**B**||)) <-- General equation

theta = acos(**A**_{n} . **B**_{n}) <-- Special case of the generalized equation (simplified because for normalized vectors **A**_{n} and **B**_{n}, ||**A**_{n}|| * ||**B**_{n}|| = 1)

**Edited by Cornstalks, 13 January 2013 - 11:08 PM.**