Programming "Widgets": how do you efficiently draw/redraw windows and widgets?

Started by
6 comments, last by SeanMiddleditch 8 years, 4 months ago

Hi.

I am wondering about this. I am curious as to how you efficiently program a "widget" GUI system, that is, a part of a program that draws interface windows with buttons and scrollbars and that type of stuff (widgets) on them.

I have done this a couple times before but I was not really satisfied with my previous attempts and am now curious about how you really make it efficient and clean. In particular, what I have is this: I represent the windows and widgets by a tree structure, which is "natural" because it has a natural hierarchical structure, in which the root node represents the whole screen, and then below that we have nodes for the windows, then each window node has nodes for the widgets, then each widget node has nodes for the components of the widget (It could go further, in theory, but this is as far as I go). The order in which the widgets/windows overlap (from front to back) is determined by the ordering of them as siblings on the tree -- the nodes keep pointers to the next sibling in a linked list.

The trouble is that I am trying to figure out how to make the render/drawing as quick and efficient as possible -- in particular, we may know that only certain widgets or components need to be redrawn, so we shouldn't need to explore every node of the tree, but we also have to deal with resolving overlap: we shouldn't redraw something with a higher position on the order if it doesn't actually overlap anything, but we also may not want to keep exploring the whole structure to check for overlaps every time we do a redraw. How can we efficiently do this whole process, including the overlap resolution, the latter ideally without having to check every sibling?

This has to have an already-existing solution since window-based GUIs are so ubiquitous. I am just not sure where to find a good description of it. What would that be?

Advertisement

Qt handles all that. If I was making a non-gaming application requiring native look and feel, I'd use Qt (which I am).

If I was making a videogame, videogames usually have rather simple GUIs, and redrawing them every frame isn't much of a cost (a hundred or so polygons). I'd only bother caching the text, because generating the text can cost alot; I'd assume anything else is premature optimization at this point in time, and only work to optimize it if it proves a bottleneck by profiling later in development.

But if I was to implement it myself, I'd just give each widget a "needsRedrawing" boolean.

It's get set to false each time it gets redrawn, and it'd get set to true when events occur that require redrawing.

For example, if a button is hovered over, the button will want to redraw itself with a hover appearance (and may want to keep redrawing itself for several frames of animation).

Or if a widget gets moved, the parent widget needs to be marked for redrawing.

If a widget is marked as 'needsRedrawing', all its children also get redrawn.

I don't understand what you're talking about with the overlap. Do you mean draggable windows that float over other windows?

I don't understand what you're talking about with the overlap. Do you mean draggable windows that float over other windows?

Overlap means yes, windows/widgets overlapping on top of each other. Not necessarily draggable -- just on top of each other, e.g. a menu and then you open up a sub-menu which floats above.

Well, for floating windows, when they appear they don't need to invalidate anything (they're drawing over it anyway).

While they are still showing, a floating window doesn't need to invalidate anything either.

Only once a floating window gets hidden again (or moves) do they invalidate.

So when a floating window gets hidden, then get its previous rect (or, if moved instead of hidden, get the rect for just the now-uncovered area), and test that rect against the widget tree. For any widget that overlaps the 'now exposed' rect, invalidate that widget, and check its children for overlap. For any widget that isn't overlapped, don't invalidate it, and you don't even need to bother checking the children.

This sort of thing was a great efficiency savings back in the days of 16-bit processors and single-digit megabytes of RAM. It's not clear it is worth worrying about these days. I'd take even odds that the cost of recursively updating all the needsRedraw flags is damn near as much as just drawing the whole thing again.

Especially when you consider that the underlying windowing system is going to submit a new frame to the GPU at least every 1/60th of a second... There's some useful caching you can do when rendering text and such, but people routinely write IMGUIs that render at 60 fps.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]


This sort of thing was a great efficiency savings back in the days of 16-bit processors and single-digit megabytes of RAM. It's not clear it is worth worrying about these days. I'd take even odds that the cost of recursively updating all the needsRedraw flags is damn near as much as just drawing the whole thing again.

I have to say you are wrong here, at least in the GUI-libary I wrote, doing this "dirty" checking resulted in enormous speedups, in the range of multiple ms on all machines. Its one of the first optimizations I would recommend when trying to improve the performance of a (non-game) GUI. It can easily be justified by how little there actually has to be redrawn regularily (nothing if you don't move the mouse cursor ie.), and the benefit from even skipping say rendering a single tree-view/attribute table already outweights any loss from keeping track of what needs to be redrawn. You might not need it to get 60 FPS, but since you want to do other stuff in your application as well, it can still save a lot of processing power for those things.

and the benefit from even skipping say rendering a single tree-view/attribute table already outweights any loss from keeping track of what needs to be redrawn

I think we may be talking slightly at cross-purposes.

Frames are going to be drawn to the display at 60hz, pretty much whether you like it or not. You can cache rendered views in GPU buffers (textures) if you don't want to rerender them for a while, and you can cache the command list used to render a set of views. But unless you have a fullscreen GUI that is entirely static (i.e. no animation), and the user stops moving the mouse... things are going to get drawn somehow.

(if you are building widgets at the OS native layer, then the windowing system is already handling all this caching for you. For example, each window on OS X is double-buffered by default, Android caches the command buffer used to render sets of views... and so forth)

When I suggest leaving out the 'needsRedraw' flag, I'm not suggesting that you re-layout all your text views every frame, or recalculate the scrolling list view's visible set every frame. Those things are expensive, and you already know exactly when these need to be done, because they are in response to external state changes. But assuming you've handled all your state changes correctly, drawing is just submitting a batch of triangles to the GPU, potentially updating some subset of those vertices to handle animations. Which is a pretty cheap operation, especially if you've cached all that in vertex buffers and/or textures already.

And the reason I press on this distinction is that modern GUIs are increasingly animated. On an iOS device, a set of depth planes are constantly rerendered to give the impression of parallax as the gyros report motion. Android's 'material design' supports (and advocates for) motion and animation on every action even indirectly triggered by user events... If you build a system to interpret drawing as the same as responding to layout, then you've largely ruled out concepts like ambient animation.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

I have to say you are wrong here, at least in the GUI-libary I wrote, doing this "dirty" checking resulted in enormous speedups, in the range of multiple ms on all machines.


Seriously. A HUGE amount of effort goes into this kind of thing on modern Web browsers (which are now also increasingly gaming platforms in their own right) and don't even get me started on how much time our UI team spends dealing with this nonsense as well (which may be more than Scaleform is terrible, but still).

This applies not just to layout but also rendering.

But assuming you've handled all your state changes correctly, drawing is just submitting a batch of triangles to the GPU, potentially updating some subset of those vertices to handle animations. Which is a pretty cheap operation, especially if you've cached all that in vertex buffers and/or textures already.


Unfortunately, interesting game GUIs are often built out of enough overlapping translucent pieces or requiring complex scissoring operations that you have to break rendering up into many many separate draw calls, which on the current generation of popular graphics APIs (OpenGL and D3D11) are still heinously expensive operations.

Doing this for a simple phone app on iOS or Android is one thing, given that the app has the whole screen/GPU to itself and generally doesn't have anything else interesting to do. Doing that for a real-time game, however, means that your UI has very little time on either the CPU or GPU to do its thing, because the game itself is already eating up so much of it.

Sean Middleditch – Game Systems Engineer – Join my team!

This topic is closed to new replies.

Advertisement