I'm rolling my own math library to reinforce my understanding of the math behind computer graphics. I think I have a good handle on vectors/matrices and their nuances (row & col vectors, matrix majorness, left & right handed rotations, left & right handed coordinate systems) and have made solid progress implementing and testing my library. However, one issue has come up that I would like some suggestions and clarifications from the good folks at GameDev.

I have opted for column vector notation so use vector post multiplication when transforming vertices. I have overloaded the * operator in my matrix class to accommodate this convention with the vector as the argument for the overloaded operator.

Now, here's what I'm fretting over. Should I overload the * operator again to allow row vector-style multiplication with the vector as the left hand argument and the matrix as the right? If so, should I implicitly transpose the matrix to allow "correct" (expected) results? Or should I overload the operator but perform a column-style post multiplication so that the user has to explicitly transpose the matrices when doing row vector-style multiplication?

Assuming that my understanding is correct and that this question isn't a result of a misunderstanding on my part, my inclination is to either not allow row vector multiplication at all (after all, I clearly state in the documentation that column vectors are assumed) or allow row vector multiplication but put the responsibility onto the user to transpose the matrices (I feel this would make the intent of any user code clearer).

I hope I have made myself clear and look forward to your suggestions.
Thanks

Your code should enforce the conventions you use. If you have column vectors, then vector-matrix multiplication is not allowed mathematically so you should not provide such an operator even if it implicitly transposes the vector and performs the corresponding multiplication. The reason is because this implicit transpose is not obvious from the code and what your code ends up looking like, is like it's a mix of row and column vectors with no specific convention.

You should make it explicit if you want to transpose the vector. For example, you can use the type system to have two vector types, and a transpose function that translated between the two. This way, you can overload operator * for row vector times matrix, and matrix times column vector.

It does open up some other issues you need to handle. For example, you may have to provide full support for two vector types (shouldn't be a problem if you're into templates), or you can simply just have a proxy type for row-vectors that only holds the vector for the purpose of introducing a new type, but does not allow any other operations. You then need to transpose the result from the multiplication back to a column vector again. For example, just to demonstrate the idea behind a second type:
[source]
struct transposed_vector {
transposed_vector(vector const &v) : v(v) {}
vector v;
};
transposed_vector transpose(vector const &v) {
return transposed_vector(v);
}
vector transpose(transposed_vector const &v) {
return vector(v);
}
transposed_vector operator *(transposed_vector const &, matrix const &);
vector operator *(matrix const &, vector const &);
[/source]
Either way, you should not implicitly convert types so that you don't, or appear not to, enforce your own conventions: if you have column vectors, then you also have a right-multiplication.

edit: I just realized that transposed_vector is probably an ambiguous name (is it row or column). Either make it a type that is not intended to be used other than as an internal type for the purpose of transposing vectors, or make both vector classes explicit about being row or column vectors.

Thanks BB, you've really cleared things up for me. I think I'll strictly enforce column vectors and not allow premultiplication. Should the user wish to use my library with row vectors then the responsibility for handling this correctly is delegated to the user. I think this would make my math library's convention clear and consistent which in turn should make the user's intent clear and consistent.

Alternatively you could just implement matrices, after all a vector is just a matrix with one row/column (see MATLAB for example)