I can't use separate mask for every element of my gui.

What CAN you do then? :)

If all you need are rounded corners, then you can generate the "masks" for those in the pixel shader, with a bit of math (by comparing the UV coordinates against the sin/cos of some corner-radius(es)). And if all your UI elements are rectangles with rounded corners, then you can also pass in the parent's width, height and corner-radius(es) for each child, and use that same math to clip the child (either discard the pixel, or multiply it's alpha value with the parent's alpha value resulting from the math). This way, you can still use quads.

If you want a more general algorithm for clipping arbitrary geometry (child elements) against other arbitrary geometry (parent elements), you can represent each element as an array of rectangles, similar to how "region" objects are represented in Win32 GDI. In fact, you could use the Region Functions to generate regions for the shapes you want, clip them against the parents' region, then send all the rectangles returned by GetRegionData to the pipeline. However, if you want to use textures for your UI elements, you'd have to add the UV coordinates to all those rectangles - that might not be hard to do, but if you ask me, it's whole lot easier to work with UV coordinates when you have everything laid out as single quads, and use masking, than to generate geometry of the exact shape you want. There might be libraries out there that can do 2D geometry (polygon) intersections for you, but I don't know of any.

EDIT: On second thought, the region functions are a bad idea, because they probably wouldn't look very good when scaled or rotated, unless you used a really high resolution for them, which would probably mean that GetRegionData would return a whole lot of rectangles, which would probably slow down other parts of your UI-processing. You should use masking.

Yeah, arbitrary geometry is probably not sensibly possible in a single pass. Same for intersecting regions. But if you're mainly aiming for round rectangles, you got me thinking. Here some food for thought doing the mask procedurally (i.e. without a texture).

Take a look at this page from Iñigo Quilez: distance functions (also search for signed distance functions). There's a formula for a rounded box. Transforming this to 2D and applying some anti-alias magic you could do parametrized round rectangles in the pixel shader:

// round rectangle distance function, taken from Iñigo Quilez. Here in 2D
float udRoundBox( float2 p, float2 b, float r )
{
    return length(max(abs(p)-b,0.0))-r;
}

float4 RoundBoxPS(float2 p: TEXCOORD) : SV_Target
{
    // parameters in texture space (0..1)
    float2 rectCenter = float2(0.5, 0.5);
    float2 rectHalfSize = float2(0.4, 0.2);
    float radius = 0.07;
    
    // take distance function (negative, we want the inside of the rect).
    float dist = - udRoundBox(p - rectCenter, rectHalfSize, radius);    

    // estimate transition for antialiasing
    float transition = length(fwidth(p));

    float mask = saturate(dist / transition);
    return float4(mask.xxx, 1);
}

Not sure if this is useful (I just like to play with 2D shaders )