Inverse trigonometric functions are applied to a length ratio which is a scalar value; there is no definition for vectors / points! With your formula you would need to calculate a division of two vectors what also is not defined that way.
Looking at the illustration in the OP, you want to compute the angle between the principal x vector
x = [ 1 0 ]T
and the vector v between point p1 and point p2
v = p2 - p1
The arccos computes the angle between the adjacent and the hypothenuse, where both of these legs are what you have given as x and v. Taking advantage of the dot-product
v . x = |v| * |x| * cos( <v,x> )
where the angle <v,x> is your theta, you get
<v,x> = arccos( ( v . x ) / ( |v| * |x| ) )
Now one of the vectors, namely x, is a unit vector, i.e. has length |x| == 1, so that the formula is simplified to
<v,x> = arccos( ( v . x ) / |v| )
Furthermore the y component of x is always 0 and its x component is always 1, so the dot-product is simplified to
v . x = vx * 1 + vy * 0 = vx
and hence the formula becomes
<v,x> = arccos( vx / |v| )
EDIT: The OP mentioned to compute "the smaller angle between the line and the x axis", which, following the more detailed requirements in one of the following posts, is too broadly interpreted by the above solution.
Edited by haegarr, 26 February 2014 - 02:11 AM.