Hmm, so you suggest to generate an alpha mask texture.
I suggest using OpenGL to generate the alpha mask texture, yes.
I guess rendering it 1/2x screen size will make it look really cheap, won't it?
Perhaps, but I think it might look good enough because your alpha would be 1/2 size, not the texture you are rendering; so the texture should disguise the lower-res alpha.
What are my other options on fading edges?
Blur, either with a pass over the mask texture itself, or by sampling multiple positions within the fragment shader when using the mask.
Here is my fragment shader:
//"in" attributes from our vertex shader
varying vec4 vColor;
varying vec2 vTexCoord;
//our different texture units
uniform sampler2D u_mask;
uniform sampler2D u_texture; //default GL_TEXTURE0, expected by SpriteBatch
void main(void) {
//sample the colour from the first texture
vec4 texColor0 = texture2D(u_texture, vTexCoord);
//get the mask; we will only use the alpha channel
vec4 mask = texture2D(u_mask, vTexCoord);
//interpolate the colours based on the mask
gl_FragColor = vec4(texColor0.rgb, mask.a);
}
Why are you using the alpha channel of your fully visible not transparent black and white image?
You want to use any channel except the alpha channel. Just use mask.r instead of mask.a, but leave it in the same position in the vec4.
You want to use the colored greyscale texture as the alpha channel of your other texture.
How can I assign a mask position, so that it's not in default bottom left corner(as I guess)? And also, how can I scale it?
[...]
I need to be able to move it somehow, e.g. to the center...
I thought you wanted the mask to fill the entire screen? If so, you don't need to position the texture - use
gl_FragCoord to index into the mask texture. You may need to normalize gl_FragCoord first. I do it like this:
//Normalize the screen coord, so we get screen coordinates from 0.0 to 1.0
vec2 viewportCoord = (gl_FragCoord.xy / ViewportSize);
//Invert the screen coordinate, because gl_FragCoord.y is the opposite of what we want (0.0 is the bottom of the screen?).
viewportCoord.y = (1 - viewportCoord.y);
Then you'd do something like:
//Use the viewport pixel positions to read from the mask, so the mask fills the entire viewport.
vec4 mask = texture2D(u_mask, viewportCoord);