Sounds like you expect the look vector to be the direction you want to view in but it is in fact the point you want to look at. I don't know if that's the case or not since I don't know the particular function you use, but that is how gluLookAt, and most other look at-functions i have seen, works.
Oh snap, you are right, I am passing in my direction vector into the values of lookX, lookY, and lookZ. And I'm passing in my position vector to the values eyeX, eyeY and eyeZ.
So I just changed it to
[source lang="java"]Matrix.setLookAtM(mViewMatrix, 0, eyeX, eyeY, eyeZ, eyeX + lookX, eyeY + lookY, eyeZ + lookZ, upX, upY, upZ);[/source]
And it works now