I think I understand what you are doing now. Instead of rotating the texture coordinates could you make a square texture where the point of the triangle starts in the centre (rather than the left). Then you rotate that quad in the desired direction.
I do believe this is to do with the aspect ratio. If you rotated the entire game area 90 degrees and then stretched it horizontal to fit the width and squashed it vertically to fit the height you would end up with a similar effect. I suppose the issue is that for textures the coordinates are 0-1,0-1, 1:1 but your screen is a much different ratio.
I also see from rereading your initial post that doing it as a rotated quad might not be possible as I first though. You could possibly use a rotated quad method with a stencil buffer but I think you might lose that faded edge on the mask. Another option might be to use render to texture and create an actual rotated mask and then use that new mask.
For your mask to be thinner when it is at 90/270 degrees the x texture coordinate range needs to be larger (between about -0.4 and 1.4 for 16:9) and you'd have to alter how the texture is setup to allow it to do that. You'd have to do something similar for the y component. It's not a neat solution :/.
I think the render to texture option would be the best choice. You just render a rotated quad with your texture on it, that creates a new mask which has already been rotated and then you use that mask instead.