I've already implemented bump/normal mapping. Now I'm trying to go back and just do diffuse without a normal map.

Everything's in tangent space, I thought. So it should be fine to do this?

	vec3 lvec = normalize(light_vec);
	float diffuse = max(dot(lvec, vec3(0,0,1)), 0.0); // Using (0,0,1) instead of normal map

But it seems the normals are in eye space, right?

	mat4 normalMat = transpose( inverse( model * view ) );
	vec3 normalEyeSpace = vec3( normalMat * vec4(normalIn, 0.0) );
	normalOut = normalize(normalEyeSpace);

And the light_vec (and lvec above) are in tangent space.

	vec3 n = normalOut; // Normal
	vec3 t; // Tangent
	vec3 c1 = cross( normalOut, vec3(0.0, 0.0, 1.0) ); 
	vec3 c2 = cross( normalOut, vec3(0.0, 1.0, 0.0) ); 

	if( length(c1)>length(c2) )
	{
		t = normalize(c1);	
	}
	else
	{
		t = normalize(c2);	
	}

	vec3 b = normalize(cross(n, t)); // Binormal

	vec3 tmpVec = sundirection;
	light_vec.x = dot(tmpVec, t);
	light_vec.y = dot(tmpVec, b);
	light_vec.z = dot(tmpVec, n);

The problem is that the shadow shifts when I rotate the camera. What do I compare/dot the lvec with in the first code to get the diffuse factor?

I think I got it. I have to calculate the normal matrix using only the model matrix, to get the normal in world(?) space.

	mat4 normalMat = transpose( inverse( model ) );
	vec3 normalWSpace = vec3( normalMat * vec4(normalIn, 0.0) );
	normalOut = normalize(normalWSpace);

you can also just supply the rotation matrix, or just use the mat3 part of the view matrix

this 'trick' requires that your view matrix is simple, as in, it's only rotated and translated

this is because you need to multiply your normal by a matrix that produces unit length vectors, and rotation-matrices (among others) do that

since your view and rotation matrix contains the rotation of your camera,

you can rotate your normal by rotating it by the mat3 part of your view matrix:

vec3 normal = mat3(matview) * in_normal.xyz; // rotate to follow camera/view rotation

you can "remove" or "undo" that rotation by multiplying with the transpose:

vec3 backToNormal = normal.xyz * mat3(matview); // rotate back again

a simpler look at creating a tangent-space matrix: