my current working solution with the depth buffer and a little discussion bonus. (Yes, I'm super excited because it's taken me a week to get here.)
Log Entry describing use of glClipPlane:
I then went on to trying with the depth buffer. I realized that neither the depth buffer or the stencil buffer allows for stacking,
Then, while searching for topics involving "glScissor", "transform", "clip" and "GUI"
I came across glClipPlane. This function which I assume is meant for space,
can set up to at least 6 clip planes, that will work like glScissor(), but are transformable.
At the time of writing, I have a small sample set up, but I've realized that
a 45deg rotated child of same size as its square parent will result in 8 edges, not 6,
and there is an infinite amount of potential edges to consider, depending on the build of the GUI tree.
So I'm abandoning this.
although I first tried to have an element pop itself once left. This actually worked out quite well:
(no depth masking, only depth testing) Render self Render children (depth masking, GL_ALWAYS) Render self with parent's depth.That's roughly pushing and popping the depth buffer. I guess the stencil buffer would work nicely like this as well.
After doing this I tried to alternate depth values (1.0, 0.0, 0.9, 0.1 | 0.5, 0.4, 0.6, 0.3) to have something to test against. I realized that with this approach,
I would still need the parent element to be the (furthest / nearest) relative to the current, to make the current clip inside the parent's boundaries.
I ended up scratching the stack behavior, and instead setting that all elements with the same element is clipped by the first,
so, if the parent's "depth" is the same as the one of the current element, the current element should test with GL_EQUAL
(Yes, I know that seems risky with floating point precision, but it works, and I wonder why it wouldn't, 0.999998 is my 1.0f everywhere.)
If the parent's "depth" is not the same as the one of the current element, the test is GL_ALWAYS.
(Then the element gets rendered on the top of everything, just like it would normally when drawing back-to-front)
It's amazing that sometimes it takes several very elaborate solutions to finally find a simple, working one.
I might edit this post later to make myself clearer...