Hello,

while continuing with implementing an OpenGL renderer to my engine, I've come across (suprisingly) the first real problem the seems to arise of the difference between the coordinate systems in DirectX and OpenGL. Up until now, I've been using the same matrices for both APIs, with a slight difference to account for the models appearently upside down in the OpenGL-version:

		Matrix MatPerspFovLH(float fovy, float aspect, float zn, float zf)
		{
			float yScale = cotan(fovy/2.0f);
			float xScale = yScale / aspect;

			return Matrix(	xScale, 0.0f,	0.0f,			0.0f,
#ifdef ACL_API_GL4
							0.0f,	-yScale,	0.0f,			0.0f,
#else
							0.0f,	yScale,	0.0f,			0.0f,
#endif
							0.0f,	0.0f,	zf/(zf-zn),		1.0,
							0.0f,	0.0f,	-zn*zf/(zf-zn),	0.0f);
		}

Flipping the sign of the 22-member in the perspective matrix seemed to fix that, so well. Seems that when using shaders the difference between the two APIs are rather marginal, regarding vertex transformation. I also had to swap the order of mul(vertex, matrix) in the shaders to matrix * vertex, but thats about it.

But once I got lighting to work again, I've noticed that the normals behave really odd. I have a scene with a tank driving around at 0.0f y-height, turning around that very axis as he drives. While the tank is turning around just as fine as before, the normals seem to turn around a completely different axis. So rather than the tank always getting lit from the suns direction, if he turns around 180° the lighting will do like a barrel roll, and now the tank is lit from the complete opposite side. I hope this is somewhat imaginable, I'll also upload a video once I've got movie-maker installed here.

In the meanwhile, I already tried everything possible that I could imagine to fix this. I tried flipping all different signs on all normals, matrices etc... involved, tried to change multiplication orders here and there, but it didn't do nothing. The real thing that throws me off is indeed the fact that the rotation of the vertices work as expected, yet the normals are totally screwed up, though I'm using the same matrix for both. Well, here is the shader I'm using:

uniform InstanceV
{
	mat4 mWorld;
};

uniform StageV
{
	mat4 mViewProj;
};

layout(location = 0) in vec3 invPos;
layout(location = 1) in vec2 invTex0;
layout(location = 2) in vec3 invNrm;

#define outvPos gl_Position
layout(location = 0) out vec2 outvTex0;
layout(location = 1) out vec4 outvPos1;

void main()
{
	outvPos1 = ( mWorld*vec4(invPos, 1.0f) );
	outvPos = ( mViewProj*outvPos1);

	outvTex0 = invTex0;

	outvNormal = normalize(( mWorld*vec4(invNrm, 0.0f)).xyz);
}

// frag_begin

layout(location = 0) in vec2 invTex0;
layout(location = 1) in vec4 invPos1;

uniform sampler2D Material;

layout(location = 0) out vec4 outvMat;
layout(location = 1) out vec4 outvPos;
layout(location = 2) out vec4 outvNrm;

void main()
{
	outvMat = texture(Material, invTex0);

	outvPos = invPos1;

	outvNrm.rgb = invNormal;
	outvNrm.rgb = normalize(outvNrm.rgb);
	outvNrm.rgb = 0.5 * outvNrm.rgb + 0.5;
	outvNrm.a = 1.0f;
}

I already simplified it so that I'm only using the world-space transformed normal instead of normal mapping. I'm actually using deferred shading here, but as you will see the problem lies in the normals themselfs already.

Regarding the used matrices, they are in the same "format" as the d3dxmath-9 ones would. Multiplication-order program-side is

Scale * Rotation * Direction * Translation

// direction is a matrix used to orient a model into a specific direction, as used to rotate my tank.

This is the code used to generate the direction-matrix:

if(Direction* pDirection = entity->GetComponent<Direction>())
{
    if(pDirection->bDirty)
    {
        pDirection->v.Normalize();

        const math::Vector3 vUp(0.0f, 1.0f, 0.0f);
        math::Vector3 vRight = pDirection->v.Cross(vUp);
        vRight.Normalize();

        const math::Vector3 vRealUp = vRight.Cross(pDirection->v).normal();

        pDirection->mRotation.Axis(pDirection->v, vRealUp, vRight);

        pDirection->bDirty = false;
    }

    mTransform *= pDirection->mRotation;
}

I quess there was a lot more code that could be helpful to show, but in between all the matrices, and other stuff I could show that probably plays a part here, I though I'd first see if somebody has got a clue before posting half of my program here. So, does anybode see an issue here? I'm really lost, and to be honest I don't have a clue what else to try, aside from some voodoo-programming by flipping around more signs and/or swap multiplication orders like crazy until maybe it works again :/ Maybe someone here already got a similar problem and solved it?

EDIT: Here is the link to a video showcasing the issue: