As I see in the video, X axis width stays the same no matter what the angle is and the same is for Y axis, height stays always the same and the mask just stretches. Maybe some multiplication in mat2() could help? I'm not even sure. Could there be something fishy with 'worldMaskSize'?
What do you mean rotate the actual quad? The only quad is a blurred map, can't really rotate it, because it has to stay in place with the map.