Hey guys, I think using OpenGL/DirectX to build GUI is a very good idea because it has hardware acceleration and advanced graphical features, which can produce a dynamic and awesome-looking GUI. However I can't find many GUI engines that are based on OpenGL/DirectX. I searched about the disadvantage of using OpenGL/DirectX to build GUI and the two disadvantages I found is: The GUI that are based on OpenGL/DirectX bypass a lot of abstract layers and access graphics card directly, which can cause issues when trying to screenshot the GUI (or windows made by the GUI) by GDI. Also, usually a window is made up of a lot of child windows, e.g. button, edit box, etc. The GUI that are based on OpenGL/DirectX don't have any child windows so the layer is invisible to outside(other programs).

I think the screenshot problem can be solved by cooperate GDI into the GUI engine (probably use GUI to draw the buffer?). For not having any child windows, I personally think it has no problem and is an advantage since you don't want other program to analyze your GUI easily.

What do you guys think? Thanks!

What are your plans for using this UI? Do you want to use it for ingame UIs, or for editors/tools? I'm asking since you are talking about "child windows". For a game, it is indeed tolerable, if not wanted, to not have a seperate window if ie. an options menu pops up. Imagine this would be a seperate window, you'd have to tab around to get back to your regular game. For editors, its usually wanted/preferred to have a seperate window pop up when you e.g. open an asset editing window. You can bypass this by creating a seperate OS window in this case, with a seperate DirectX-device, and rendering to this instead, if it ever becomes an issue, so this should not be a showstopper. However, if you are planning on really using this for editor-like applications and not for ingame graphics, I'd strongly suggest you use an already existing libary like QT, since the complexity required for building a custom-rendered UI capable of simulating all the widgets needed for editors (window docks, tables, trees, ...) is overwhelming. For ingame UIs, there is nothing wrong with writing the UI renderer yourself, and in this case you should not have an issue with that everything is "one element" and cannot be inspected from outside.

Regarding the screenshot problem, I can't really follow. Whats the issue again? I have no problem making screenshots of my own DX/OpenGL-renderer UI. In fact if this was a problem than screenshotting wouldn't work in games at all, which it does 100% of the time. Well thinking about it, I think OpenGL on windows didn't work, but thats a different story... you don't need to fallback to GDI in that case eigther, since you can always just capture the content of the backbuffer at the end of thre frame and output that, just search for the different ways this can be done via google.

What are your plans for using this UI? Do you want to use it for ingame UIs, or for editors/tools? I'm asking since you are talking about "child windows". For a game, it is indeed tolerable, if not wanted, to not have a seperate window if ie. an options menu pops up. Imagine this would be a seperate window, you'd have to tab around to get back to your regular game. For editors, its usually wanted/preferred to have a seperate window pop up when you e.g. open an asset editing window. You can bypass this by creating a seperate OS window in this case, with a seperate DirectX-device, and rendering to this instead, if it ever becomes an issue, so this should not be a showstopper. However, if you are planning on really using this for editor-like applications and not for ingame graphics, I'd strongly suggest you use an already existing libary like QT, since the complexity required for building a custom-rendered UI capable of simulating all the widgets needed for editors (window docks, tables, trees, ...) is overwhelming. For ingame UIs, there is nothing wrong with writing the UI renderer yourself, and in this case you should not have an issue with that everything is "one element" and cannot be inspected from outside.

Regarding the screenshot problem, I can't really follow. Whats the issue again? I have no problem making screenshots of my own DX/OpenGL-renderer UI. In fact if this was a problem than screenshotting wouldn't work in games at all, which it does 100% of the time. Well thinking about it, I think OpenGL on windows didn't work, but thats a different story... you don't need to fallback to GDI in that case eigther, since you can always just capture the content of the backbuffer at the end of thre frame and output that, just search for the different ways this can be done via google.

Thank you for replying.

I am planning for editors/tools. I guess I didn't express what I meant clear enough, sorry about that.

What I meant by child windows are window components like buttons and edit box(they're child windows in MFC). I personally think it's better just to render the components, instead of creating child windows for them. Actually, what is point of that? That allows other programs to easily analyze my GUI. What do you think?

What I meant by screenshot is the case when other programs try to screenshot my GUI, such as remote desktop. Some of those program uses GDI to screenshot so..

Are there any disadvantage if I use OpenGL to make GUI for editors/tools? I think the two disadvantages I mentioned above are not really disadvantages. What do you guys think?

What I meant by child windows are window components like buttons and edit box(they're child windows in MFC). I personally think it's better just to render the components, instead of creating child windows for them. Actually, what is point of that? That allows other programs to easily analyze my GUI. What do you think?

As I said, I think it is perfectly fine to render buttons, etc.. directly instead of creating seperate window elements for them. I wouldn't worry about the other way around though, whats the problem with programs analyzing your UI? The one advantage over having seperate windows is if you really have a window that pops up, like an asset editor, and you want the user to be able to tab between it and your editor. THEN is the time to pop up a seperate window.

But again, if you are planning on using this for an editor, I once again like to urge my opinion: DON'T. I did it myself, and its a path of no joy. Do you want to do this as a learning experience? Thats is fine, in this case keep going. Do you want to be productive and create a future-rich editor in no time? Then stop with this approach, and just get Qt. You'll have most of your editor in like 6 months compared to 2 years of pain. I did this myself, and yeah, I still to this day sometimes cringe over the bugs and flaws of the UI system, and having to write a new sort of advanced widget type is always a pain (next up would be allowing windows to be reinserted into the docking structures for me, ugh). Don't worry about factors like external applications being able to analyze your UI then, its a non-issue, since literally every other UI application is built that way, only very few do their UIs themselfes (Unreal Engine it appears, but their manpower/knowledge behind this is also much better).

Are there any disadvantage if I use OpenGL to make GUI for editors/tools? I think the two disadvantages I mentioned above are not really disadvantages. What do you guys think?

To put more emphasise on it, the main disadvantage is that its awefully unproductive, even if you need to learn QT, its like 4-16 times faster. Learning a QT widget is like maximum of few hours of reading the (good) documentary and trying it out, writing the same widget can take days, and you'll spend even more time fixing bugs, and polishing. Again, if you are in for the learning experience, its fine, but don't expect to gain anything from doing it other than that.

What I meant by screenshot is the case when other programs try to screenshot my GUI, such as remote desktop. Some of those program uses GDI to screenshot so..

I still don't think you have to worry about this. Remote desktop manages to capture all 3D game applications so far (I once was foolish enough to try playing highend games over the internet on my phone, lol). The point is, its regardless of what those applications use, they will most likely capture the content of the framebuffer, and don't even care what application is visible, so you don't need to care eigther. If in doubt, try it. Just render a colored quad in OpenGL and pop in your target capturing program, and see if it works (it should).

What I meant by child windows are window components like buttons and edit box(they're child windows in MFC). I personally think it's better just to render the components, instead of creating child windows for them. Actually, what is point of that? That allows other programs to easily analyze my GUI. What do you think?

As I said, I think it is perfectly fine to render buttons, etc.. directly instead of creating seperate window elements for them. I wouldn't worry about the other way around though, whats the problem with programs analyzing your UI? The one advantage over having seperate windows is if you really have a window that pops up, like an asset editor, and you want the user to be able to tab between it and your editor. THEN is the time to pop up a seperate window.

But again, if you are planning on using this for an editor, I once again like to urge my opinion: DON'T. I did it myself, and its a path of no joy. Do you want to do this as a learning experience? Thats is fine, in this case keep going. Do you want to be productive and create a future-rich editor in no time? Then stop with this approach, and just get Qt. You'll have most of your editor in like 6 months compared to 2 years of pain. I did this myself, and yeah, I still to this day sometimes cringe over the bugs and flaws of the UI system, and having to write a new sort of advanced widget type is always a pain (next up would be allowing windows to be reinserted into the docking structures for me, ugh). Don't worry about factors like external applications being able to analyze your UI then, its a non-issue, since literally every other UI application is built that way, only very few do their UIs themselfes (Unreal Engine it appears, but their manpower/knowledge behind this is also much better).

Are there any disadvantage if I use OpenGL to make GUI for editors/tools? I think the two disadvantages I mentioned above are not really disadvantages. What do you guys think?

To put more emphasise on it, the main disadvantage is that its awefully unproductive, even if you need to learn QT, its like 4-16 times faster. Learning a QT widget is like maximum of few hours of reading the (good) documentary and trying it out, writing the same widget can take days, and you'll spend even more time fixing bugs, and polishing. Again, if you are in for the learning experience, its fine, but don't expect to gain anything from doing it other than that.

What I meant by screenshot is the case when other programs try to screenshot my GUI, such as remote desktop. Some of those program uses GDI to screenshot so..

I still don't think you have to worry about this. Remote desktop manages to capture all 3D game applications so far (I once was foolish enough to try playing highend games over the internet on my phone, lol). The point is, its regardless of what those applications use, they will most likely capture the content of the framebuffer, and don't even care what application is visible, so you don't need to care eigther. If in doubt, try it. Just render a colored quad in OpenGL and pop in your target capturing program, and see if it works (it should).

Thank you very much for your full and detailed answer. I'll consider about that! :)

I am planning for editors/tools.

Then use the platform's native windowing toolkit, so that your app acts like a good citizen.

What I meant by child windows are window components like buttons and edit box(they're child windows in MFC).

MFC has been superseded at least four times: ATL, WTL, Windows Forms, WPF. It should not be an input constraint on your decision making.

I personally think it's better just to render the components, instead of creating child windows for them. Actually, what is point of that? That allows other programs to easily analyze my GUI. What do you think?

Don't be paranoid. Use the platform native windowing toolkit.

Getting GUI behavior correct is challenging, and if you go with a non-native GUI method for a windowed non-game application, you will constantly remind your users that you got the behavior wrong. Can you guarantee that you will properly support clipboard? What about tab order? Keyboard shortcuts? And you want to do all this engineering because of the imagined threat that a screenshotting application will "use GDI" to analyze your GUI?

Use the platform native windowing toolkit.

Are there any disadvantage if I use OpenGL to make GUI for editors/tools? I think the two disadvantages I mentioned above are not really disadvantages. What do you guys think?

It's a huge waste of time because modern platform native windowing toolkits already use accelerated graphics libraries like Direct3D (Windows) and OpenGL (Mac, now moving to Metal). You won't even see a performance benefit for all your work.

Use the platform native windowing toolkit.

It's a huge waste of time because modern platform native windowing toolkits already use accelerated graphics libraries like Direct3D (Windows) and OpenGL (Mac, now moving to Metal). You won't even see a performance benefit for all your work.

Its even way worse than that. By manually rendering the UI, you will tie your editors performance to the game "views" performance. So if your game runs very slowly at some point (debug mode, debug information like terrain-wireframe/collision boxes/etc..., unoptimized 4k-texture bloat by artists), your editor will also run as slow and become extremely unresponsive. I have the same issue, and as far as all the information I've been able to gather, the only way to solve this via heavy multithreading ground up, so that your editor and game basically run in their own thread. I don't have to mention that this is a huge overkill in terms of how you will have to design your entire application.

I really forget to mention this point, so thanks for Oluseyi for bringing it up.

Its even way worse than that. By manually rendering the UI, you will tie your editors performance to the game "views" performance. So if your game runs very slowly at some point (debug mode, debug information like terrain-wireframe/collision boxes/etc..., unoptimized 4k-texture bloat by artists), your editor will also run as slow and become extremely unresponsive. I have the same issue, and as far as all the information I've been able to gather, the only way to solve this via heavy multithreading ground up, so that your editor and game basically run in their own thread. I don't have to mention that this is a huge overkill in terms of how you will have to design your entire application.

I really forget to mention this point, so thanks for Oluseyi for bringing it up.

Cheers.