Hi!

I'm trying to rotate a mask in my shader code:

void main(void)
{
vec4 texColor0 = texture2D(u_texture, vTexCoord);

vec2 worldPos = vec2(vPosition);

vec2 maskTexCoord = vec2((worldPos.x - worldMaskPos.x) / (worldMaskSize.x), (worldPos.y - worldMaskPos.y - worldMaskSize.y) / (-worldMaskSize.y));

maskTexCoord = maskTexCoord + vec2(-0.5, -0.5);
maskTexCoord = mat2(cos(aimAngle), sin(aimAngle), -sin(aimAngle), cos(aimAngle)) * maskTexCoord;
maskTexCoord = maskTexCoord - vec2(-0.5, -0.5);

vec4 mask = texture2D(u_mask, maskTexCoord);

gl_FragColor = vec4(texColor0.rgb, mask.r);

}

Not sure why, but it get's scaled as it gets closer to 90 degree angle. Angle 0 degree:

Angle 90 degree:

Any ideas what's wrong?

I'm not entirely sure, your -translation, rotation, +translation part looks ok so I would think the error hides here:

vec2 maskTexCoord = vec2((worldPos.x - worldPlayerPos.x) / (worldMaskSize.x), (worldPos.y - worldPlayerPos.y - worldMaskSize.y) / (-worldMaskSize.y));

I have no idea what's going on in there though.

Can you explain what some of your values are. What exactly are you rendering? What shape, is it a triangle?

There is a mask texture, that is a quad. It basically calculates the UV coords for the mask texture. Note, that if the angle is 0, it works fine, so I suspect something fishy with the rotation part.

Your rotation matrix does look fine, instead of using the angle to create it use

mat2(0, 1, -1, 0)

That is a 90 degree rotation matrix so if you get the same issue using that then I think you can rule out the matrix being the problem.

I assume when you are using it properly you set the angle from code? float uniformAngle = 3.14 / 2.0;

Can you show the mask texture? is that quad full screen? Trying to better understand your method.

I changed my mask texture, so that it's longer and saw, that it doesn't get scaled with even 0 angle correctly. This line was causing it:

worldPos.x -= worldPlayerPos.x - vPosition.x;

I made it like this:

vec2 worldPos = vec2(vPosition);

I updated the original shader code in first post.

However, the rotation problem persisted. Here is a mask texture:

I tried replacing matrix calculation with yours provided, nothing changes. I made a quick video for you to be able to understand better:
https://vid.me/Xgj3

I wonder if it has anything to do with the aspect ratio.

Would it not be easier to rotate the actual quad instead of using the texture coordinates?

As I see in the video, X axis width stays the same no matter what the angle is and the same is for Y axis, height stays always the same and the mask just stretches. Maybe some multiplication in mat2() could help? I'm not even sure. Could there be something fishy with 'worldMaskSize'?

What do you mean rotate the actual quad? The only quad is a blurred map, can't really rotate it, because it has to stay in place with the map.

I think I understand what you are doing now. Instead of rotating the texture coordinates could you make a square texture where the point of the triangle starts in the centre (rather than the left). Then you rotate that quad in the desired direction.

I do believe this is to do with the aspect ratio. If you rotated the entire game area 90 degrees and then stretched it horizontal to fit the width and squashed it vertically to fit the height you would end up with a similar effect. I suppose the issue is that for textures the coordinates are 0-1,0-1, 1:1 but your screen is a much different ratio.

I also see from rereading your initial post that doing it as a rotated quad might not be possible as I first though. You could possibly use a rotated quad method with a stencil buffer but I think you might lose that faded edge on the mask. Another option might be to use render to texture and create an actual rotated mask and then use that new mask.

For your mask to be thinner when it is at 90/270 degrees the x texture coordinate range needs to be larger (between about -0.4 and 1.4 for 16:9) and you'd have to alter how the texture is setup to allow it to do that. You'd have to do something similar for the y component. It's not a neat solution :/.

I think the render to texture option would be the best choice. You just render a rotated quad with your texture on it, that creates a new mask which has already been rotated and then you use that mask instead.

I fixed it, thank you! All I did was just change my mask to be a square:

Basically, just added some empty space. Seems to be a really scammy solution, but hey, at least it works

I fixed it, thank you! All I did was just change my mask to be a square:

Basically, just added some empty space. Seems to be a really scammy solution, but hey, at least it works

Brilliant, can't fault a solution like that :D.