After a day of searching for answers I've found out that there seem to be 2 ways of doing this.

1. Calculate some kind of texel area of the triangle and divide this by the triangle area. (What exactly is meant by "texel area" and how do I calculate it using my texCoords[0,1] ?)

2. This: (GLSL code)

float mipmapLevel(vec2 uv, vec2 textureSize)
{
  //rate of change of the pixels in u and v with respect to window space
  //approximate to au/ax, au/ay, av/ax, av/ay
  vec2 dx = dFdx( uv * textureSize.x);
  vec2 dy = dFdy( uv * textureSize.y);
  
  //select the LOD based on the maximum compression of an edge in texture space.
  //This corresponds to the maximum length of a side in texture space
  //max (sqrt(dUdx*dUdx + dVdx*dVdx),
  //    sqrt(dUdy*dUdy + dVdy*dVdy));
  float d = max( dot (dx, dx), dot( dy, dy));
  
  //convert d length to power-of-two level of detail
  return 0.5*log2(d);
}

I've been trying to get my head around these partial derivatives the entire day but I still don't quite get it.

The best I've come up with was something that "kind of" works but I'm sure it's not entirely correct:

I calculate the change of the texCoords between the 2x2 rasterized/shaded area:

float ddx = abs(uv_X.m128_f32[1] - uv_X.m128_f32[0]);
float ddy = abs(uv_Y.m128_f32[2] - uv_Y.m128_f32[1]);

And then get the mip level like so:

ddx *= texSizeX;
ddy *= texSizeY;

Vector4_SSE vec0 = Vector4_SSE(ddx, ddy, 0, 0);
float d = 0.5f * log2f(Vec4_SSE_Dot(vec0, vec0));
return std::max<unsigned int>(std::min<unsigned int>(static_cast<int>(d), 8), 0);

What I just don't understand is why the formular says au/ax, av/ax, au/ay, av/ay. Isn't the rate of change in window coordinates always going to be exactly 1 pixel ?? Doesn't this make the ax and ay variables completely void ? Or I'm just not getting it so please enlighten me

Hmm okay but what you're talking about is the rate of the texel's change in window coordinates which is the result of the above forumlar, right ?

What I don't get is what ax and ay relates to in my code.

If au and av is the rate of change from one pixel to the neighboring (UV[x+1, y]-UV[x, y] and UV[x, y+1]-UV[x,y]), then ax and ay would be the rate of change of the window coordinate. But wouldn't this always be 1 from one pixel to the neighboring ?

edit: or does the equation only mean that it is au in relationship to ax ? And so can be ignored in the code (the divide) ? Still..something is wrong in my code.

What I'm doing is calculating for the entire 2x2 quad: abs(p(1,0) - p(0,0)) and abs(p(0,1) - p(0,0)) since I thought the rate of change is an absolute value that should be the same for all x and y in the 2x2 quad. Is that correct ?

Edit: I assume the part I'm missing is only for non-uniform textures ? (e.g. 128x64 instead of 128x128)

If I understand correct I also need to multiply the derivatives of X and Y both by the texWidth and both by texHeight and then do the max(dot(xx,xx), dot(yy, yy)) right ?

I think I've got it working now