Hi everyone!

Since I've began implementing my GUI I've known that I wanted to be able to cut away parts of gui elements that are outside the boundaries of the parent element.

I've always just thought: "I'll deal with that through stencils". But now I've found that my approach may be more difficult than so.

In this example I have 3 elements to draw (I'll end up not knowing about the number of levels or elements). On the first level, I have one rectangle in which I'd like the contents to be cut if they exceed its bounds.
(Familiar to CSS overflow:false) I also have a triangle which is placed somewhere close to the edge of the screen.

Inside the rectangle, on the next level, I have a circle which lies just on the boundary of the rectangle. It should be "cut in half" once rendered.

It looks like this when rendered:

The rectangle is the only element with "stenciling = true", painting and enabling the stencil buffer before its children are drawn.

As the Triangle is drawn after the rectangle. (right after the circle actually, when using a regular recursive approach where an element is rendered, then it's children)

It is completely invisible due to the stencil "created by" the rectangle.

If the circle had its stenciling enabled rather than the rectangle, the circle itself would be completely visible, but the Triangle would still be invisible.

I hope this explains my situation.

As possible solutions, I've thought of passing a rectangle to the element fragment shader, allowing discards when the element is outside the bounds. -That might actually be a better approach...

Something else I thought of, still using stencils, is rendering a layer of children sequentially, and then rendering the following layers afterwards (sort of defying the tree structure I have created)

I intend to be able to rotate elements as well, which I think seems doable, and something else that would be really cool with this stencil sort of option is if the

child elements would have their fragment alpha multiplied by the ones of the parent in the given screen space coordinate.

This does seem very elaborate for a "simple gui", but I'm sure I can find a way to make it work, although I might have to change the way I accomplish things.

If you have any suggestions or experience with this, you're encouraged to tip in!

EDIT: I typed "I have 3 elements to draw" which lead responders to think that it's the only scenario (obviously, because that's what I said!)

Unfortunately, I won't have so much knowledge about the elements and transforms involved...

- You could simply only draw what should be seen, like only half of the circle. That would avoid needless work that gets cut anyway.
- You can ignore what would be outside the screen as it would get cut away anyway(or if easier just not draw part of it).
- You can use the scissor test if you have a rectangular area where you draw in like with the circle.
- You could use the depth buffer for if you for example would draw the half circle first and later the rectangle a bit behind it and avoid overdraw there.
- You can use the stencil buffer, but you need to have one at first and then set some config for it and set it to the needed pattern when needed for each item (like deactivating the test before the triangle), which is more complicated than the other choices.

In the end its probably best to go the easiest way for you first and only optimize if there really is a problem with performance.

Why don't you just set the scissort test to match your parent x,y,width,and height. then anything that falls outside that bound will not be drawned.

Hi guys,

Thank you for taking the time to respond, I appreciate it, although the solution might require a little more effort.

a) - You could simply only draw what should be seen, like only half of the circle. That would avoid needless work that gets cut anyway.
b) - You can ignore what would be outside the screen as it would get cut away anyway(or if easier just not draw part of it).
c) - You can use the scissor test if you have a rectangular area where you draw in like with the circle.
d) - You could use the depth buffer for if you for example would draw the half circle first and later the rectangle a bit behind it and avoid overdraw there.
e) - You can use the stencil buffer, but you need to have one at first and then set some config for it and set it to the needed pattern when needed for each item (like deactivating the test before the triangle), which is more complicated than the other choices.

f) In the end its probably best to go the easiest way for you first and only optimize if there really is a problem with performance.

a: That would be great. I better make a draw call to the entire circle, though. But I like the way you see the problem, though.

b: The screen? Yes. A rectangular shape with n-levels of transforms (S,R,T)? No. I cannot ignore it entirely. Hence, my original post.

c: That's a good suggestion. I actually hadn't used glScissor() until today, as I never tried to pull this sort of thing off before. It seems it's bound to the viewport, though.

I'd really need a way to do glScissor() while using the gl transform stack. (Some unknown grandchildren can be transformed, and they might need to use glScissor() too...)

d: Using the depth buffer is actually a really good idea. I think I might end up using that. It's just a shame because i thought the stencil buffer could pull off what i need.

I need overdraw on all of the elements, though. I have no idea what elements have transcluency and which ones that don't. :-(

e: I think I should get back to this once before deciding on the depth buffer approach.

f: I fully agree! Thanks for all your input, it's much appreciated!

Why don't you just set the scissort test to match your parent x,y,width,and height. then anything that falls outside that bound will not be drawned.

I think I might have worded my question wrong, but there's no set tree or set amount of elements or levels. If the circle had its own child element,

a pentagon which is placed in the far right side of the circle. (outside the bounds of the blue rectangle), this approach would cause the pentagon to be rendered.

(Only because glScissor() is not inherited through the gl transform stack as it seems [please tell me if this isn't correct]) On the top of that, I don't think glScissor() can be transformed at all.

But for this specific problem, your approach would work.

Usually you would have a setup where one toplevel window has several childwindows of which some have grandchildwindows and so on in a treelike structure. Then you set the scissor according to the parent window each window got and that should cut away anything outside of allowed bounds.

With the depth buffer you would draw the child windows first and then the parent, so where the child has put something there is no need to draw the parent. Transparency would be only all or nothing when doing it that way and you would then also activate the alpha test as to avoid writing to the depth buffer on transparent points and later draw the parent window there. If theres half transparent parts you could draw the parent window first and then combine the color from framebuffer with the child window or try to avoid this with premultiplied alpha.

The pentagon wouldnt be rendered if the scissor is set smaller than the rectangle. You would probably need to constantly keep track of how it is set and only shrink it for child windows when some grandparent window already set it smaller than the parent window.

If part of that pentagon was inside the rectangle and outside the circle and you want to cut that part away (or similarly for other nonrectangular shapes of (grand-)parent window) that could necessitate using stencil.

You would probably need to constantly keep track of how it is set and only shrink it for child windows when some grandparent window already set it smaller than the parent window.
If part of that pentagon was inside the rectangle and outside the circle and you want to cut that part away (or similarly for other nonrectangular shapes of (grand-)parent window) that could necessitate using stencil.

Exactly. I guess I'll have to read more into stencil buffers, then. The depth map thing seems a little too elaborate in comparison to what i need done. Perhaps I should just aim for that all elements are allowed to be rendered outside their parent...
It's not trivial, that's certain. It would be cool if I could have a "stencil" buffer stack where I could draw a mask for every window using AND, and pop it back to the previous state. when I need to render the rest of the parent's siblings.

Without yet looking further into stencil buffers, I'm currently looking into glClipPlane, which I've used with volumes, but not in the plane.

It also seems that there've been two other similar gamedev posts:

http://www.gamedev.net/topic/363996-problem-with-glscissors/

http://www.gamedev.net/topic/498511-multiple-clipping-planes/

Although they're quite dated, they seem like they'd be an easy way to set some arbitrary rectangular boundaries.

There's still the matter of nesting multiple layers of rectangular holes.

I don't mean to be using this as a development blog; but if you guys have some experience in 2d plane clipping, fire away!

For now, I'll execute random experiments on my GUI tree! :D

For people who in the future would like a solution to this problem, I will include my log entry describing one more attempt on solving it,
my current working solution with the depth buffer and a little discussion bonus. (Yes, I'm super excited because it's taken me a week to get here.)

Log Entry describing use of glClipPlane:

Then, while searching for topics involving "glScissor", "transform", "clip" and "GUI"
I came across glClipPlane. This function which I assume is meant for space,
can set up to at least 6 clip planes, that will work like glScissor(), but are transformable.
At the time of writing, I have a small sample set up, but I've realized that
a 45deg rotated child of same size as its square parent will result in 8 edges, not 6,
and there is an infinite amount of potential edges to consider, depending on the build of the GUI tree.
So I'm abandoning this.

I then went on to trying with the depth buffer. I realized that neither the depth buffer or the stencil buffer allows for stacking,
although I first tried to have an element pop itself once left. This actually worked out quite well:

(no depth masking, only depth testing)
Render self
Render children
(depth masking, GL_ALWAYS)
Render self with parent's depth.

That's roughly pushing and popping the depth buffer. I guess the stencil buffer would work nicely like this as well.
After doing this I tried to alternate depth values (1.0, 0.0, 0.9, 0.1 | 0.5, 0.4, 0.6, 0.3) to have something to test against. I realized that with this approach,
I would still need the parent element to be the (furthest / nearest) relative to the current, to make the current clip inside the parent's boundaries.

I ended up scratching the stack behavior, and instead setting that all elements with the same element is clipped by the first,
so, if the parent's "depth" is the same as the one of the current element, the current element should test with GL_EQUAL
(Yes, I know that seems risky with floating point precision, but it works, and I wonder why it wouldn't, 0.999998 is my 1.0f everywhere.)

If the parent's "depth" is not the same as the one of the current element, the test is GL_ALWAYS.
(Then the element gets rendered on the top of everything, just like it would normally when drawing back-to-front)

Very simple!
It's amazing that sometimes it takes several very elaborate solutions to finally find a simple, working one.
I might edit this post later to make myself clearer...