clue why it is happening?

Got the same issues a few years back, so read with care. There might be a better/correcter solution

If I remember correctly , then you have trouble with changing the handness of your transformation matrix. Sure, the surface normal is still point outside your model (which is needed for the light calculation), but the "normal" of the transformation matrix should be pointing inside when mirrored. Practically the green lines on the left side should point inside the model to be correct.

This leads to an second issue: when do you need to invert the normal from the transformation matrix to get the correct surface normal ? Well, I have saved it in as sign of an else always positive value (I use a quaternion to represent the tangent space matrix) to encode the "handness".