Greetings.

I'm working on GUI. I have a parent-child relationship and I want to make parent "maskable", i.e. every child pixel outside parent should be discarded. The problem is that I render my GUI in single pass and can't figure out how to use stencil (I think it's not possible in my case). Also my elements are semi-transparent.

I render all elements without depth test - I submit geometry in order - from deepest to nearest. And I'm trying to achieve the following:

I numerated the quads by order. 1 - deepest one, next 2, 3 is a child of 2 and should be clipped, 4 is above 2 (and 3 since it's a child). All quads are semi-transparent.

As you can see - stenciling will not help here. I tried different approach - every element have an id. Every child also have parent id. In fragment shader when I render a parent I write to a RWTexture an id. Next when I render a child - I check if a RWTexture have a parent id. If no - discard pixel. That worked amazingly well! But here http://www.gamedev.net/topic/665588-does-gpu-guaranties-order-of-execution/ clever guys told me that pixel shader executes in random order and writes to RWTexture happens randomly. So it's possible that child will try to write before parent (thought I never see this in my tests). And I can't rely any more on this technique. And right now I don't have any ideas how to achieve clipping in single draw call.

P.S.: the layout and number of element is not known at compile time. Also elements can be transformed and have round corners so there's no way to analytically calculate child pixel position relative to parent.

I use the scissor-test in OGL for this (don't know if there's something similar in DX). Nevertheless, you could implement this in a shader, just discard the pixels which are not inside the bounding rect of the parent element.

But here http://www.gamedev.net/topic/665588-does-gpu-guaranties-order-of-execution/ clever guys told me that pixel shader executes in random order and writes to RWTexture happens randomly.

Take a look at what MJP wrote:

When you draw triangles, the value output from the pixel shader (using SV_Target) will be written to your render target in the order of triangle submission.

For this case, only the per pixel case is important.

Nevertheless, you could implement this in a shader, just discard the pixels which are not inside the bounding rect of the parent element.

If you have only axis-aligned rectangles you can simply clip them when filling up your vertex buffer.

As I wrote in PS, I have rotating quads. Also quads can have round corners (and maybe holes) so it will be difficult to calculate it analytically. It was my mistake that I wrote about quads. I probably have to write arbitrary geometry.

For non-rectangular regions, you could use a second texture as a mask and do manual blending (so to speak) in the pixel shader. This can of course be combined with the above approach.

This sounds interesting. Can you give more insights please?

You'll probably need stenciling for this.

Stenciling will not work in my case. Please take a look at a problem description.

The mask can be a single channel texture

What is this mask? How do I get get/create it?

Thank you very much. The code is very interesting, especially vertex and uv geneartion. But unfortunatelly I can't see how I can apply this technique to my case. I can't use separate mask for every element of my gui.