Designing a GUI system, hints appreciated.

Started by
100 comments, last by Shinkage 13 years, 9 months ago
Quote:Original post by swiftcoder
And how is this functionally any better in a RMGUI? Sure, the artists can add a button to an RMGUI in the GUI designer, but until the programmer writes a function for the button to call, it doesn't do anything.


Exactly - the artist can lay out the visuals without programmer intervention. In fact, an artist can add twenty buttons, sprites popping in an and out, and replace the 'face' of the UI without needing the programmer at all. To make the buttons do something the programmer needs to step in, of course, but ideally a simple visual change would not require a code change.

With IMGUI once the layout & code are tied together, an artist can't simply add/remove buttons/sprites without modifying the code. So the artist, who really only cares about visuals, now needs to have intimate knowledge of the underlying code as well. The separation between logic & presentation does not seem clear enough to allow either to work completely independently.

Now, granted, a tool could potentially merge the presentation and logic together allowing the programmer and artist to work independently, but I haven't seen anything like this. Most IMGUI's I've seen are very programmer-friendly (script file editing and such), and not very artist friendly.


Quote:Original post by swiftcoderAnd that brings us to the meat of the issue: why in hell is your artist adding buttons willy-nilly? The artist doesn't know what functionality is required, that would be the programmer's responsibility.


Usually artists and programmers, working together, have a close understanding of what's needed on screen. They will discuss the need to have a button here, a sprite there, a window yonder. The artist can start laying out screens w/ the needed buttons and whatever else and get it ready for the programmer to tie it all together.

Similarly, the programmer can work ahead based on an agreement between himself and the artist, knowing that (at some point) a screen will come with a button named "PLAY_GAME" or whatever, and when that button is pressed, the game will launch. Functionality can be built independently from the presentation, and vice versa. But, as I said, it does require that the artist and programmer are in agreement with what to expect in the UI.

Quote:Original post by swiftcoder
With an IMGUI, the programmer figures out functionality, stubs out the GUI hooks *beforehand*, and then the artist modifies the layout and themes GUI based on the stub.


You can do the exact same with RMGUI. A programmer can literally fire up a tool, lay out some buttons and windows that he expects, hook up the functionality, and tell the artist to go hog-wild. I don't see where IMGUI has a strength in that regard ;) It's a process change, not a code/system change.

Quote:Original post by swiftcoderThe exact same way an RMGUI does: you run a timer, which updates certain attributes of the GUI elements.


Coded by hand or does a tool do that? I don't think there's any reason NOT to have a tool do the bulk of an IMGUI system, but that the UI performs layout and logic simultaneously makes me curious how it would be done without turning an IMGUI solution into an RMGUI-like system.
Advertisement
Quote:Original post by Grafalgar
Quote:Original post by swiftcoder
With an IMGUI, the programmer figures out functionality, stubs out the GUI hooks *beforehand*, and then the artist modifies the layout and themes GUI based on the stub.
You can do the exact same with RMGUI. I don't see where IMGUI has a strength in that regard ;) It's a process change, not a code/system change.
I never said it was a particular strength of IMGUI, but nor do I see it as a weakness.

There is also nothing stopping you from producing an IMGUI designer tool which allows the artist to add buttons, etc. and then produces the stubbed-out source code - just like all RMGUI designer tools do currently.

Quote:
Quote:The exact same way an RMGUI does: you run a timer, which updates certain attributes of the GUI elements.
Coded by hand or does a tool do that?
The *exact* same way an RMGUI does. I even use the same source files between my older RMGUI and an IMGUI.

The only fundamental difference between an IMGUI and an RMGUI is the way in which elements are specified, and the way in which state is propogated to/from the application. Most of the IMGUIs you see are trivial examples, and thus don't implement the full feature set of an RMGUI - but they aren't in any way representative of any limitations of the method.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

Quote:Original post by Grafalgar

Exactly - the artist can lay out the visuals without programmer intervention. In fact, an artist can add twenty buttons, sprites popping in an and out, and replace the 'face' of the UI without needing the programmer at all. To make the buttons do something the programmer needs to step in, of course, but ideally a simple visual change would not require a code change.

With IMGUI once the layout & code are tied together, an artist can't simply add/remove buttons/sprites without modifying the code. So the artist, who really only cares about visuals, now needs to have intimate knowledge of the underlying code as well. The separation between logic & presentation does not seem clear enough to allow either to work completely independently.


The simplicity and separation of concerns that you describe here are not a property of IMGUI vs RMGUI but rather the fact that your GUI is static.

A lot of UIs aren't that simple. They're dynamic. They have controls that operate on data in the app. They have widgets that appear and disappear depending on the state of the app. They have lists and hierarchies of widgets that reflect the structure of data in the app.

These are inherent properties of the UI, and you can't shield an artist from them. You can free them from having to work in code, but they still have to understand concepts like "there will be one of these panels for each item in this list" or "this panel will be displayed whenever this condition is true."

This is all true regardless of whether your GUI library has an IM or RM interface. I think the reason that you associate properties of static UIs with RMGUI is that RMGUI is really only good for dealing with static UIs.

With RMGUI, you give the library an initial definition of the UI, and the library holds onto it, assuming it will stay the same. If you want any dynamic behavior, you have to go out of your way to change the UI definition inside the library.

IMGUI APIs are designed for dynamic UIs. You write a function that gets called each frame to look at the state of your application and produce the UI for that frame. And you can use all the same control structures you normally use, like if statements and for loops, which map quite naturally to the dynamic properties I described above. In other words, instead of giving it a static definition of the UI, you give it parameterized definition that will yield different UIs depending on the state of the application.

Unfortunately, when a lot of people watch Casey's presentation, they seem to miss this point and come away with other less important points, like some of the assumptions he made in his particular implementation.

So yes, the simplest way to implement it is to have presentation and functionality tied together in the code. That's true for RMGUI as well, but it's not inherent to either concept. In either case, you can still get presentation information from external specifications and have tools for manipulating it.

Anyway, for your case, if you're willing to limit yourself to static UIs with only buttons, labels, and sprites, then the two approaches are essentially equivalent (the parameterized definition would have no parameters!), so I wouldn't worry about it, unless you're interested in eventually supporting more.
Quote:Original post by swiftcoder
Quote:Original post by Grafalgar
How does IMGUI handle complex animations? I admit that I am not intimately familiar with the approach aside from the handful of examples I've seen.
The exact same way an RMGUI does: you run a timer, which updates certain attributes of the GUI elements.

Simple IMGUI's (like the nvidia sample) obviously can't do this, because they don't persist any state behind the scenes, but an IMGUI of any complexity may retain just as much information as a RMGUI - the key element being that the user doesn't have to deal with it.
I couldn't figure out a way to do this, unless IDs are guaranteed to be "very unique". I think however Grafalgar has a point here because we're putting state in a IMGUI which was supposed to be almost state free or left management to the application. The paradigm probably does not work equally well for complex widgets. I feel like it would be violating some kind of unwritten rule, conceptual integrity maybe. Driving those attributes from outside the GUI manager doesn't just feel right to me.
I cannot see how to justify existance of a modifiable data blob for a control which I am guaranteeing to be living for a frame. I just don't yet feel right for me. Maybe I could declare in advance some special widgets to be retaining state, but this sounds too much RMGUI to me, I'm not sure.

I find no much of an issue in providing more complicated backends, as far as I can just fire doControl and forget about it. If the state is internally managed, that would be ok, but it's quite easy to require more. Just require a button to pulsate on hover and suddendly its state changes. Scripted GUI maybe? It probably makes sense at this point to just go retained mode.

Considering the design a bit more in detail, I see an IMGUI can be massively complicated. Layering must be done with consistently less information, for example. I start to think that "complex" IMGUIs are probably not worth the effort, although they can still be extremely useful in the low-profile GUI field, very likely far better than any RMGUI system.
I am still positive presentation can be separated, but I am not yet well aware of the degree of the benefits. It probably allows only minor changes to occur, such as some theming.
Quote:Original post by MoundS
A lot of UIs aren't that simple. They're dynamic. They have controls that operate on data in the app. They have widgets that appear and disappear depending on the state of the app. They have lists and hierarchies of widgets that reflect the structure of data in the app.
...
This is all true regardless of whether your GUI library has an IM or RM interface. I think the reason that you associate properties of static UIs with RMGUI is that RMGUI is really only good for dealing with static UIs.
I have to agree with this, I needed to do that for a quasi-contract I got some months ago and the pain involved in "dynamicizing" the GUI was absolutely surprising.

Previously "Krohm"

Quote:Original post by Krohm
I couldn't figure out a way to do this, unless IDs are guaranteed to be "very unique". I think however Grafalgar has a point here because we're putting state in a IMGUI which was supposed to be almost state free or left management to the application. The paradigm probably does not work equally well for complex widgets. I feel like it would be violating some kind of unwritten rule, conceptual integrity maybe. Driving those attributes from outside the GUI manager doesn't just feel right to me.

I cannot see how to justify existance of a modifiable data blob for a control which I am guaranteeing to be living for a frame. I just don't yet feel right for me. Maybe I could declare in advance some special widgets to be retaining state, but this sounds too much RMGUI to me, I'm not sure.


IMGUI libraries can retain state behind the scenes. It's not a violation of any IMGUI principle. It's only a violation if you force the application to manage it. And of course, if you can retain state, you can do anything an RMGUI library can do (animations, etc).

The only problem is figuring out which data goes with which widget. I've found that trying to manually associate a unique ID with each widget is too big of a burden to place on the application developer, so my library automatically figures out the data associations using a combination of widget order (inside basic blocks), macros to track control flow, and IDs to keep track of lists where the items can change order. You can read a lot about this topic on the Molly Rocket IMGUI forum.

Quote:Considering the design a bit more in detail, I see an IMGUI can be massively complicated. Layering must be done with consistently less information, for example. I start to think that "complex" IMGUIs are probably not worth the effort, although they can still be extremely useful in the low-profile GUI field, very likely far better than any RMGUI system.


Not sure what you mean by this. I agree that writing an IMGUI library is more complex than writing an RMGUI library, because an RMGUI library just gives up on trying to manage things and forces the application to do it. But that just forces the application code to be that much more complex. Why solve the problem over and over in every app, when you can just solve it once in a library?

Quote:I am still positive presentation can be separated, but I am not yet well aware of the degree of the benefits. It probably allows only minor changes to occur, such as some theming.


If you go the extra step of assigning text labels to your widgets (at least, locally unique labels), then you can do more than just theming. Layout order doesn't necessarily have to be the same as execution order. But if they're different, then you need a way of matching them up. So, yes, you would lose some of the benefits of IMGUI, but it would still be a lot better than full-fledged RMGUI.
A while ago I came across the Adobe Source Libraries and Adam and Eve, which are property model and layout libraries respectively, and uses declarative languages to specify the gui components. It's certainly pretty interesting from a design point of view (albeit the libraries are not being very actively developed atm).

Has anyone else come across these / have any view on this type of design?
Quote:Original post by sprite_hound
A while ago I came across the Adobe Source Libraries and Adam and Eve, which are property model and layout libraries respectively, and uses declarative languages to specify the gui components. It's certainly pretty interesting from a design point of view (albeit the libraries are not being very actively developed atm).

Has anyone else come across these / have any view on this type of design?


I've seen a video presentation of it that was very good, but unfortunately I can't find it anymore.

It looks like they're focused on data binding issues. A lot of those issues are solved by IMGUI anyway, since widgets directly work with application data, and you can sync up two widgets by just having them refer to the same application data. Adam & Eve seems to go a little further. You can say things like "constrain these two values to the same proportions if this check box is checked". But I suppose you could implement that in a layer on top of IMGUI.

What I'm curious about (and what I didn't see in the presentation), is how Adam & Eve handles the dynamic issues I mentioned above. If it doesn't address those, I couldn't see using it for serious work.
For what it's worth, I implement my GUI completely the same as normal game objects. This means that they end up using the exact same material system as my other rendered objects, allowing complete customization - anything the engine can render can be done for a GUI too. All of the logic is handled by controllers being added to the objects that will respond to picking queries, click events, UI events and whatever else.

Try to find a way to do this - then all of your scripting tools, game editor tools, and any other tools that work with your game objects automatically can be applied to your GUI... Plus then you have no problem making a 3D GUI instead of a standard 2D version, and you have much less code to maintain since you are reusing everything!
I'm generally against automatic layout. However, it obviously depends on your use case. The last GUI I designed and implemented was for a fixed resolution screen (on a CE device), so automatic layout probably wouldn't have gained me any time. I'm a bit of a perfectionist when it comes to placing GUI components too, so I would have explicitly defined positions relative to borders and other components in any case.

The most important pattern I have in my code is a Listener/Observer pattern. It's very simple to implement and really is the only sensible way to do things with a GUI, as it keeps spaghetti to a minimum.

Bigger picture:

However, I find that the most important thing you can do before actually writing any code whatsoever is to design your GUI on paper. That is you should know pretty much 75-90% of everything your GUI is going to be expected to do and only then should you start to think about the components you'll need and how to display them (obviously people will chip in with new requirements/changes, hence you can never make it 100% - although big corporations try to, which is why any minor change usually takes them 6 months!).

In my last project I started with a requirements document that listed and numbered each requirement (I grouped them into logical sections according to what they did). From each requirement, I designed a GUI layout, broadly showing what each component would do/how it would work. This document ran to 80 or so pages and contained one screen layout for each dialog, plus a few for general layout and a description of how the basic components worked from a user interaction point of view (list box, scroll-bar, etc.).

The great thing about a design document is that is shows you've thought about what you're trying to do (!) and it allows other people to come in and critique or point out things you've left out or that may not work from an interaction point of view. As your design is mostly graphical, it's easy for others to visualise how the software will work too, which is a big bonus over just a list of "things that will happen" when you interact with component A or B, etc.

Once the requirements and GUI design is done, you can get to work on actually developing your GUI components. I was lucky in that I had a clever guy working on the framework. He produced for me a simple windowing system, a modified AGG graphics library and a modified freetype font library that we kind-of developed in parallel with the actual application.

From the design document, it was easy to produce the GUI, because I didn't have to think too much about how to do things (I'd already thought about it). The project came in early. My manager said in the technical meeting when it was signed off that it was the first time in 20 years he could sit there and say everything was done on time!

So if I were to give advice about GUI development, it's to design the user interaction side first, rather than jump in thinking about the technical side from the off.






Thank you all for your posts. I took some time to think at what I should add right now, I considered expanding on this or that concept, but I won't as that would probably end up killing the discussion.
What I did in those hours was experimenting with the implementation I had produced and it's ability to scale. I still cannot trust it so I went back here and to MollyRocket's forums. I only had a quick glance at them previously, but with my code getting grosser and grosser, I couldn't just walk along. I originally hoped to at least be able to communicate my decisions, but I realize this is just impossible, as I'm still very confused. I will try to pour in some thoughts with some degree of separation.

Use multi-frame GUI!


So far, I've tried my best to work without the single-frame lag. I didn't do this because of the time lapse involved, I just didn't consider it useful. However, as I previously noted, I'll need to buffer the various draw requests anyway to guarantee some minimal performance scalability - with at least one drawcall per widget, things would have turned slow FAST - so right now alot of this machinery is already in place. Long story short, I was wrong and didn't correctly evaluate benefits and drawbacks.

I was thinking about rewriting towards a dual-frame GUI however, as you'll read below, I am now considering bite the bullet and go RMGUI instead.

Maybe finding nv-widget wasn't good


I was very impressed with this library because with a simple implementation and no IDs to generate everything felt like a dream.
However, I overlooked the fact that as icastano notes: "The NVIDIA IMGUI does not use IDs, but instead it requires widgets to stay at the same screen location to retain the focus. It's not ideal, but works fine for our purposes and keeps the code simple". I'm not truly sure of what he meant and I'll save you from the (probably useless) flow of thoughts that resulted but I now think the library made me overly optimistic. I am quite on the opposite line of thinking now, as I'm not even sure to be able to break this.

By the way, while at this I just want to link to Zero Memory Widget, maybe someone will be able to see the light.

ID generation: not for the faint of heart


I stopped reading at about half 2006 and I suddenly figured out that after all, given the amount of messages in four years, maybe IMGUI just isn't as well understood as I originally believed. To be completely honest, I now suspect the paradigm likely didn't deliver as expected. I can understand that many people like me will have discussed the trouble in their forums of choice instead of on MollyRocket's but it seems there just have been not so much traffic, isn't it?

In particular state retaining requires some kind of ID-ing which must be automatic - I agree with MoundS when he says that it's too much of a burden to place on library's user - and it turns out that ID generation is still being discussed.
Even worse than that, I'm now a bit confused about auto IDing. In general, state retaining seems to be VERY complicated.

Current line of thinking


Today I will redesign to reach a few different goals. First of all, IMGUIs are not about efficiency but on ease of use. I see a parallel with scripting and I have some script-ish machinery which could probably support the problem with some slight modifications. So, right now, I foresee different IMGUI systems.
The IMGUI I had developed up to now will become the native C++ IMGUI system. It will be nowhere as complex at originally anticipated. No complex widgets, little to no state retaining, no eyecandy, no theming.
Its only goal is to reach sufficient sophistication to support components on top of it and even this is now open to debate. I hope to never use this directly, if not in the initial stages.

I then considered a managed, script-assisted IMGUI system. This is quite more lengthy so it's discussed below.

The third system would be a managed, script assisted RMGUI. I hoped that I could delay this to the far future but with more and more doubts arising I am considering just biting the bullet. As I've started to think seriously at this only a few hours ago, I'm not even sure how far I want to go with this. Sadly, the very same problems of dynamic IMGUIs apply there so how to link the systems (GUI<->application) is not quite clear to me besides having callbacks. Years ago I used GLUI, which allowed to pour data directly in application's varables and it looked quite nice to me. I thought that maybe reflection could somehow help. Say for example that I pass a structure with all the variables I need to fetch and the system figures out the layout himself... It's a very blurred picture for the time being.

Managed, script only IMGUI


I considered this to be my goal for a couple of days as it promised enough functionality to be a robust backbone. Ideally, the managed IMGUI system will be the preferred way of doing things for a long time. Being based on scripts as I said, I can jump thuru a few hoops, and one of that is automatic ID generation. I was thinking at automatically generated IDs like
{    thisRef    callOff    {        groupIndex?        appResPtr?    }}

where

  • thisRef is the "this" pointer from which the doControl call originated - who created that control and owns it.

  • callOff comes from VM magic and it's doControl CALL instruction's offset in the "code segment". It is, for all purposes a sub-line call identification, __LINE__ on steroids, where the control was created. This still doesn't quite work in loops, nor in recursive things, that's why I have

  • { groupIndex?, appResPtr? }: to discriminate controls born in the very same call, I've considered either a sequential group ID or storage id. Neither seems to work to me.
    The group index would be set at 0 with a BeginGroup call and essentially count the widgets in the group until a corresponding EndGroup call is issued.


Let me build an example. Suppose I want to model a shop in which I can mark the items (say swords and shields) to buy. Say the list of swords and shields can change, even during interaction based on some metric. Suppose I draw some kind of themed checkbox which draws the object, a green tick and pulsates the background a-la win7 (so the "pulse" is the button-specific, uninteresting state). When done, the user goes checkout.
I'd like to just do:
...for(int i = 0; i < sword.length; i++)    doCheckBox(buySword, ...); // because I have no choice but saving that myselffor(int i = 0; i < shield.length; i++)    doCheckBox(buyShield, ...);if(doButton("Check out")) CheckOut(sword, buySword, shield, buyShield);...

Stuff like this will break {thisRef, callOff} big way as it cannot figure out the loop. With a sequential index, I could make it work anyway: if I can find a {thisRef, callOff} pair then I check the other parameters. There's a big issue here. If I want to hash by slot result, then I'm effectively asking the memory to stay persistent. This is all wrong.

Now, suppose in frame N I have 5 swords and 3 shields. In the following frames, I mark buySword[3] and buyShield[0]. Say it takes us to frame M.
At frame M+1 sword[3] becomes unavailable (say some designer wanted the sword to be available only at 20:00, with time going on while interacting. The very same code will be executed and what happens is that

  • If I am ID-ing on storage, then I'm ... in trouble. There's no way the app can mark the state as invalid. For first, it would admit there's state. What will happen is that GUI-internal state will be moved to the wrong button. The buy tick will move to the button originally associated with (frame N) buySword[4].
    This is more generally a very troublesome issue: no matter how smart the ID generation algorithm is, I just cannot see how it could deal with data shuffling, and this is a big problem.
    Hashing on "source data pointer" (as opposed to destination) just does not feel right, it would imply for example that I cannot change a button's text description without invalidating its state. That's a no-go either.

  • If I'm IDing on group index then I'm free of reallocating the buyXXX arrays which seems way more reasonable, but in this scenario, the very same thing happens.


RMGUIs do not solve the above issues, but it seems there's light at the end of the tunnel: they say since the beginning there will be state to manage.

In "ID Generation and data lifetime" (link above) tmadden notes in a very well thought message that "The key [for dynamic IMGUIs] is that IDs don't identify individual widgets, they identify entire trees of static code.", he proposed a way to deal with this involving an anchor and a dynamic_block. I still have to fully digest his message, but it looks it's just the object-oriented version (admittedly correct and exception-safe) than my BeginGroup and EndGroup calls.
Nonetheless, I feel like requiring the machinery he's suggesting coul be so complex one could just go for a standard RMGUI at this point!


So, summing up, even making up my mind and rethinking my goals, I am now very confused.
I seriously appreciate what burnhand wrote, it essentially builds down to "keep it real". The issue is that I cannot even tell if the examples I make up in my mind do have sense, if the GUI I have in mind can work. The fantasy shop example above will break pretty much everything I can think at.
This is terrible.

Previously "Krohm"

This topic is closed to new replies.

Advertisement