I'm trying to rotate a view direction vector in my shader into tangent space. I need this for cubemap reflections to work on an arbitrary normal mapped surface. However for some strange reason the resulting reflection is rotated by 90 degrees in the YZ plane. It used to work correctly until I introduced the tangent space rotation. I suppose that the rotated viewDir vector is rotated incorrectly by 90 degrees. What's the reason and how can I fix it?

Here's a snippet from my CG shader:


v2f vert (appdata_tan v)
{
v2f o;
o.pos = mul (UNITY_MATRIX_MVP, v.vertex);

float3 binormal = cross( v.normal, v.tangent.xyz ) * v.tangent.w;
float3x3 rotation = float3x3( v.tangent.xyz, binormal, v.normal );
o.viewDir = mul (rotation, ObjSpaceViewDir(v.vertex));
}
fixed4 frag (v2f i) : COLOR
{
half3 refl = reflect( normalize(i.viewDir), half3(0.0, 1.0, 0.0) );
refl.x = -refl.x;
texcol.rgb = texCUBE( _Cube , refl ).rgb;
return texcol;
}


it looks like your reflection vector is in tangent-space rotated viewspace, which is extremely wrong
the reflection vector to a cubemap is in local-model space, meaning 0.0, 1.0, 0.0 is up etc.
to use it, you simply need to (for example):
l_reflect = reflect(-v_eye, v_normal) * mat3(camera_rotation);
this is assuming that v_eye is normalize(-position.xyz) in viewspace, and v_normal is the normal in view space

Note2: I'm using the term "matrot" here, so that you will understand we are talking about the "normal matrix"
which is the rotation part of a unit-scale view-matrix (or camera if you will) with the look-vectors
If your model AND view-matrix are simple: rotation and translation (no scaling), then you can use the 3x3 of matview

NOTE: in GLSL vector * matrix = transpose(matrix) * vector

another way:
l_reflect = reflect( (-v_eye) * mat3(matrot), in_normal );

a third way:
l_reflect = reflect( normalize(position.xyz) * mat3(matrot), in_normal );


so, bottom line here is that the reflection vector must not be rotated even if the model which represents uses the cubemap is
because you are accessing the cubemap texture using a vector that is aligned to the cubemap, not any objects!
also: you don't have to normalize cubemap accesses! so avoid normalization!


once you have the texture, you'll want to rotate the normal:
// in fragment / pixel shader
// tangent space normal:
vec3 norm = cubeMap(blabla).xyz * 2.0 - 1.0;

// you have already created your 3x3 tbn matrix in your own code correctly
// rotate your normal from tangent to localized space:
norm = normalize( tbn * norm ); // you usually don't need to normalize this

// rotate local to view-space, where matrot is the 3x3 part of a simple view-matrix (no scaling)
vec3 v_norm = mat3(matview) * norm;

or
vec3 v_norm = myNormalMatrix * norm;

done.

You're right, reflection vector should be in world space, and not in tangent space.