HLSL mul() and row/column major matricies in directx

Matrix row/column major property doesn't imply specific storage layout. Well, when speaking about math, and not about computer language.

Perhaps you're using some math lib that stores matrices by columns, or maybe you've generated matrices initially transposed.

If I understand your description, you're assuming that you can generate the transpose of a matrix product only by reversing the order of multiplication. However,

N.B.: Projection*View*World is NOT equivalent to (World*View*Projection)^T

(World * View * Projection)^T = Projection^T * View^T * World^T, that's the product of the transposes in reverse order (EDIT: not the transpose of the product), and is not equivalent to Proj * View * World.

I.e., P^T * V^T * W^T is what you want to use in the shader, but, even using the transpose of Proj * View * World results in W^T * V^T *P^T - not the same.

Btw, see this: http://msdn.microsoft.com/en-us/library/windows/desktop/bb509634%28v=vs.85%29.aspx

HLSL uses row-vector column-major math by default. I.e. it expects matrix to be column-major packed in order to do transform in most efficient way, with dot instructions.

I am using xnamath, and I think this stores its matrices in row-major order. my view of w*v*p = (w*v*p)T is wrong when wondering if you could multiply pvw * vertex using column major. I am wondering now, why do I have to transpose the wvp matrix before sending to hlsl vertex shader? how does hlsl mul() function work, that is, how is mul(vertex, wvp) different from mul(wvp, vertex)? thanks

why do I have to transpose the wvp matrix before sending to hlsl vertex shader?

Because hlsl by default expects column-major packed matrix, and not row-major.

Having single column in one constant register allows to quickly calc resulting vector component with one dot instruction.

Single mul(row-vector, column-major-matrix) will be calculated as:

    dp4 oPos.x, v0, c0
    dp4 oPos.y, v0, c1
    dp4 oPos.z, v0, c2
    dp4 oPos.w, v0, c3

while doing mul(column-major-matrix, row-vector) will be calculated as:

    mul r0, v0.y, c1
    mad r0, c0, v0.x, r0
    mad r0, c2, v0.z, r0
    mad oPos, c3, v0.w, r0

You can change default behavior with fxc options /Zpr, /Zpc.

/Zpr will inform hlsl compiler that matrices will be packed in row-major order, and you will have to to mul(row-major-matrix, column-vector)

how does hlsl mul() function work, that is, how is mul(vertex, wvp) different from mul(wvp, vertex)?

If you google for "hlsl mul function," you, too, can find this link.

Btw, see this: http://msdn.microsoft.com/en-us/library/windows/desktop/bb509634%28v=vs.85%29.aspx

HLSL uses row-vector column-major math by default. I.e. it expects matrix to be column-major packed in order to do transform in most efficient way, with dot instructions.

How can I send in the WVP matrix, not having to transpose it?

How can I send in the WVP matrix, not having to transpose it?

You can either specify /Zpr when compiling shader, or add "row_major" type modifier to wvp variable.
In both cases you will have to rewrite shader to multiply matrix by vector, and not vector by matrix.

Mathematical row/column major defines how you logically arrange your data - do you put the basis vectors in logical columns or rows.

Mathematical "row major" matrix:
x1,x2,x3,0
y1,y2,y3,0
z1,z2,z3,0
p1,p2,p3,1

Mathematical "column major" matrix:
x1,y1,z1,p1
x2,y2,z2,p2
x3,y3,z3,p3
0, 0, 0, 1

When you choose between these, it determines whether you'll be writing:
vecD = vecC * matrixA * matrixB
Or
vecD = matrixB * matrixA * vecC

Computer Science row/column major defines how you store arrays in memory.
Given the data-driven
ABCD
EFGH
IJKL
MNOP

A comp-sci "row major" storage order is:
ABCDEFGHIJKLMNOP

A comp-sci "column major" storage order is:
AEIMBFJNCGKODHLP

The storage order has no impact on the maths at all. If youre changing the maths because of the comp-sci majorness, then something is wrong.


HLSL uses comp-sci column-major array storage by default (but you can override this with the row_major keyword).
HLSL does not pick a mathematical convention for you - that's just determined by the way you do your maths.

On the C++ side, the maths library that you're using will determine both the array storage convention and the mathematical convention.
What math library are you using?

On the C++ side, the maths library that you're using will determine both the array storage convention and the mathematical convention.
What math library are you using?

I am using xnamath and it's matrices. I believe they are stored row-major because I must transpose them before sending them to the vertex shader. Now, in my vertex shader I have to use mul(vertex, matrix) because with mathematical "row-major" you go from left to right that is, vertex * world * view * projection. I know I must transpose the wvp because hlsl accepts matrices in its column-major storage.

hodgeman, buckeye, vstrahk, thnks for your help. I'm pretty sure I get it now. i used row_major in front of my float4x4 WVP matrix. i was able to send in my WVP matrix and have the mul(vertex, WVP) work just fine. I was also able to do mul(PtVtWt, vertex) and that worked just fine. PtVtWt being the reversed order matrix if you multiply with a column vector which mul(matrix, vector) does.