Return the zero vector.
You are trying to scale A (which is the zero vector) by an indeterminate dot(A, B) / dot(A, A). dot(A, B) == 0, dot(A, A) == 0, so you are trying to calculate A * (0/0), but A is zero anyway => return A (which is the zero vector).
To see this (non-rigorous argument) is correct, consider the limit when length(A) tends to zero. (EDIT: If you halve the length of A, the length of the projected vector is halved too, so it is easy to see that the limit when length(A) -> 0 is the zero vector).
EDIT: Considering the limit is the way to do it, but it doesn't apply in all cases e.g. normalisation of a vector, which has no limit as the length tends to zero. You have to consider each function separately.
Edited by Paradigm Shifter, 20 August 2013 - 03:58 AM.
"Most people think, great God will come from the sky, take away everything, and make everybody feel high" - Bob Marley