I have my WVP matrix = World * View * Projection in direct3D. If directx stores its matrices in row major style, and the hlsl mul() function can have mul(vertex, matrix) as row major style multiplication, then why must I tranpose the directx matrix before sending to the hlsl vertex shader file? also, how can I have directx send in column major matrices and have those properly multiply out? I've tried WVP matrix = Projection * view * world and then sent that into the vertex shader which uses mul(wvp, vertex) which should be column major and it doesn't work!

Matrix row/column major property doesn't imply specific storage layout. Well, when speaking about math, and not about computer language.

Perhaps you're using some math lib that stores matrices by columns, or maybe you've generated matrices initially transposed.

Btw, see this: http://msdn.microsoft.com/en-us/library/windows/desktop/bb509634%28v=vs.85%29.aspx

HLSL uses row-vector column-major math by default. I.e. it expects matrix to be column-major packed in order to do transform in most efficient way, with dot instructions.

I am using xnamath, and I think this stores its matrices in row-major order. my view of w*v*p = (w*v*p)T is wrong when wondering if you could multiply pvw * vertex using column major. I am wondering now, why do I have to transpose the wvp matrix before sending to hlsl vertex shader? how does hlsl mul() function work, that is, how is mul(vertex, wvp) different from mul(wvp, vertex)? thanks

why do I have to transpose the wvp matrix before sending to hlsl vertex shader?

Because hlsl by default expects column-major packed matrix, and not row-major.

Having single column in one constant register allows to quickly calc resulting vector component with one dot instruction.

Single mul(row-vector, column-major-matrix) will be calculated as:

    dp4 oPos.x, v0, c0
    dp4 oPos.y, v0, c1
    dp4 oPos.z, v0, c2
    dp4 oPos.w, v0, c3

while doing mul(column-major-matrix, row-vector) will be calculated as:

    mul r0, v0.y, c1
    mad r0, c0, v0.x, r0
    mad r0, c2, v0.z, r0
    mad oPos, c3, v0.w, r0

You can change default behavior with fxc options /Zpr, /Zpc.

/Zpr will inform hlsl compiler that matrices will be packed in row-major order, and you will have to to mul(row-major-matrix, column-vector)

Btw, see this: http://msdn.microsoft.com/en-us/library/windows/desktop/bb509634%28v=vs.85%29.aspx

HLSL uses row-vector column-major math by default. I.e. it expects matrix to be column-major packed in order to do transform in most efficient way, with dot instructions.

How can I send in the WVP matrix, not having to transpose it?

How can I send in the WVP matrix, not having to transpose it?

You can either specify /Zpr when compiling shader, or add "row_major" type modifier to wvp variable.
In both cases you will have to rewrite shader to multiply matrix by vector, and not vector by matrix.