Since I've began implementing my GUI I've known that I wanted to be able to cut away parts of gui elements that are outside the boundaries of the parent element.
I've always just thought: "I'll deal with that through stencils". But now I've found that my approach may be more difficult than so.
In this example I have 3 elements to draw (I'll end up not knowing about the number of levels or elements). On the first level, I have one rectangle in which I'd like the contents to be cut if they exceed its bounds.
(Familiar to CSS overflow:false) I also have a triangle which is placed somewhere close to the edge of the screen.
Inside the rectangle, on the next level, I have a circle which lies just on the boundary of the rectangle. It should be "cut in half" once rendered.
It looks like this when rendered:
The rectangle is the only element with "stenciling = true", painting and enabling the stencil buffer before its children are drawn.
As the Triangle is drawn after the rectangle. (right after the circle actually, when using a regular recursive approach where an element is rendered, then it's children)
It is completely invisible due to the stencil "created by" the rectangle.
If the circle had its stenciling enabled rather than the rectangle, the circle itself would be completely visible, but the Triangle would still be invisible.
I hope this explains my situation.
As possible solutions, I've thought of passing a rectangle to the element fragment shader, allowing discards when the element is outside the bounds. -That might actually be a better approach...
Something else I thought of, still using stencils, is rendering a layer of children sequentially, and then rendering the following layers afterwards (sort of defying the tree structure I have created)
I intend to be able to rotate elements as well, which I think seems doable, and something else that would be really cool with this stencil sort of option is if the
child elements would have their fragment alpha multiplied by the ones of the parent in the given screen space coordinate.
This does seem very elaborate for a "simple gui", but I'm sure I can find a way to make it work, although I might have to change the way I accomplish things.
If you have any suggestions or experience with this, you're encouraged to tip in!
EDIT: I typed "I have 3 elements to draw" which lead responders to think that it's the only scenario (obviously, because that's what I said!)
Unfortunately, I won't have so much knowledge about the elements and transforms involved...