There's a lot going on here that I am not qualified to comment on but whether a matrix is 'row major' or 'column' major doesn't affect the maths. there are no row/column major matrices. That comes purely just from how they are arranged in memory but there are tricks you can do such as multiplying in reverse order and transposing and stuff which does work because of how they are laid out in memory but it is far too confusing for me.

And second problem. Why is that correct output.Pos = mul(inPos, wvp); And this is wrong?? output.Pos = mul(wvp, inPos); HLSL uses column matrices, and as far as i know from theory, to transform a vector you multiply matrix * vector. Could someone explain what is goin on here? What am i missing in the matrix mess?

The reason you can't do this is because the 2 matrices don't match up, the number of columns in the first must be the same as the number of rows in the second. I guess the vector inPos is seen as a matrix that is 4 columns and 1 row. The matrix is presumably 4x4 so the #columns of the first matches the #rows of the second. With the second version the number of columns in the first is still 4 but the number of rows in the second is now only one so you can't do it. I don't know if it is true that inPos is seen as a matrix with 4 columns and 1 row but that's the only way you could have it at the start of the multiplication with a 4x4 matrix. I think things being colum/row major just confuses matter but for a vector it is quite important to know if it is seen as a single row or a single column and that can be independent of how it is laid out in memory.

The matrix you use if you have a row vector rather than a column vector would be a transpose of each other to actually give you the results you wanted.

Disclaimer, I don't use HLSL I am just looking at that as a mathematical point of view.

Well, i know a lot of linear algebra, the matrices, transpositions, multiplication order etc. I suspect hlsl does something implicitly i dont know about...

I think it all comes down to how it sees vectors. I always see them as columns but I guess (judging from the order you had to use when multiplying) that it sees them as rows. Compared to GLSL where my vectors are always the second param to a multiplication with a matrix suggesting they are seen as columns there.

There's a lot going on here that I am not qualified to comment on but whether a matrix is 'row major' or 'column' major doesn't affect the maths. there are no row/column major matrices. That comes purely just from how they are arranged in memory

Yes, and no :)
The HLSL column_major/row_major keywords do just define how the data is arranged in memory, and the XMMatrix* functions do use row_major memory arrangement, but there is also a mathematical concept.
e.g. given two 2x2 matrices containing two basis vectors, A and B:

?     ?    ?     ?
?Ax Ay? vs ?Ax Bx?
?Bx By?    ?Ay By?
?     ?    ?     ?

The first one stores basis vectors in rows, so you could say it's using row-major math.
The second one stores basis vectors in columns, so you could say it's using column-major math.

When inspected as a 1D array in memory:
The first (row-major math) in row-major storage order is the same as the second (column-major math) in column-major storage order: Ax, Ay, Bx, By.
The first (row-major math) in row-major storage order is the same as the second (column-major math) in column-major storage order: Ax, Bx, Ay, By.

When writing out your matrix code, the storage order should mostly be invisible to you. The only time you should care about storage order is when getting two different matrix libraries (or languages -- XM<->HLSL in this case) to talk to each other and share data correctly.
However, the mathematical convention (whether the basis vectors are in the rows or columns) is a major concern to how you write out your code. Opposite conventions will have opposite multiplication orders.

But yep, the OP is mixing up these two different concepts. Your XM and HLSL code may be using different storage order, but that is irrelevant to you. It's the mathematical conventions that dictate the correct multiplication order to use - and you have not switched mathematical conventions in HLSL.

For example, let's say you're using row-major mathematical conventions, and have this matrix:

?     ?
?Ax Ay?
?Bx By?
?     ?

XM uses row-major storage, so it expects this to be stored in memory as: Ax, Ay, Bx, By.
HLSL uses column-major storage, so it expects it to be stored in memory as: Ax, Bx, Ay, By, which is why you have to transpose your data before giving it to HLSL - purely because of the 1D storage order convention.
However, both XM and HLSL are both still working with the same mathematical matrix as drawn above in 2D. That mathematical matrix has not been transposed!
On the other hand, if you failed to transpose your data before handing it to HLSL, the mathematical matrix would become transposed, because HLSL would be interpreting the data incorrectly :lol:

There's a lot going on here that I am not qualified to comment on but whether a matrix is 'row major' or 'column' major doesn't affect the maths. there are no row/column major matrices. That comes purely just from how they are arranged in memory

Yes, and no :)
The HLSL column_major/row_major keywords do just define how the data is arranged in memory, and the XMMatrix* functions do use row_major memory arrangement, but there is also a mathematical concept.
e.g. given two 2x2 matrices containing two basis vectors, A and B:

?     ?    ?     ?
?Ax Ay? vs ?Ax Bx?
?Bx By?    ?Ay By?
?     ?    ?     ?

The first one stores basis vectors in rows, so you could say it's using row-major math.
The second one stores basis vectors in columns, so you could say it's using column-major math.

When inspected as a 1D array in memory:
The first (row-major math) in row-major storage order is the same as the second (column-major math) in column-major storage order: Ax, Ay, Bx, By.
The first (row-major math) in row-major storage order is the same as the second (column-major math) in column-major storage order: Ax, Bx, Ay, By.

When writing out your matrix code, the storage order should mostly be invisible to you. The only time you should care about storage order is when getting two different matrix libraries (or languages -- XM<->HLSL in this case) to talk to each other and share data correctly.
However, the mathematical convention (whether the basis vectors are in the rows or columns) is a major concern to how you write out your code. Opposite conventions will have opposite multiplication orders.

But yep, the OP is mixing up these two different concepts. Your XM and HLSL code may be using different storage order, but that is irrelevant to you. It's the mathematical conventions that dictate the correct multiplication order to use - and you have not switched mathematical conventions in HLSL.

For example, let's say you're using row-major mathematical conventions, and have this matrix:

?     ?
?Ax Ay?
?Bx By?
?     ?

XM uses row-major storage, so it expects this to be stored in memory as: Ax, Ay, Bx, By.
HLSL uses column-major storage, so it expects it to be stored in memory as: Ax, Bx, Ay, By, which is why you have to transpose your data before giving it to HLSL - purely because of the 1D storage order convention.
However, both XM and HLSL are both still working with the same mathematical matrix as drawn above in 2D. That mathematical matrix has not been transposed!
On the other hand, if you failed to transpose your data before handing it to HLSL, the mathematical matrix would become transposed, because HLSL would be interpreting the data incorrectly :lol:

Thanks for clarifying that. I did notice that to swap between using a row/column vector required transposing the transform to get the correct answer on my little scrap of paper that I checked with.

It turns out that mul(a,b) does b*a and not a*b...

You mean the HLSL mul function? No, it does compute a*b.
If it really did compute b*a, then the "//ok" lines below would give errors, while the "//error" lines would give truncation warnings:

	float2x4 _2x4;
	float4x2 _4x2;
	float4x3 _4x3;
	float3x1 _3x1;
	float2x2 result1 = mul(_2x4,_4x2);//ok
	float4x4 result2 = mul(_4x2,_2x4);//ok
	float2x3 result3 = mul(_2x4,_4x3);//ok
	float4x4 result4 = mul(_4x3,_2x4);//error - 4x3 * 2x4 isn't defined
	float4   result5 = mul(_4x3,_3x1);//ok
	float3x3 result6 = mul(_3x1,_4x3);//error - 3x1 * 4x3 isn't defined