In general, you would define a shape at the bottom of the screen (say, a rectangle) and then dedicate specific parts of it to do what you want. So in the panel, you have a rectangle in the lower left which loads a portrait when a unit is clicked, a set of shapes that respond when clicked for command buttons, and so on.

So you'll have to define the screen areas, draw the shapes, set up event hooks to populate the shapes with the correct pictures when a unit is clicked, and set up event hooks to carry out the right commands when buttons are clicked. It's not really any different than setting up any other graphics on the screen that the player can interact with, and if you've ever coded a menu or any graphical interface you've probably already got the techniques that you need.

Sit down and work out a design. UI code can quickly become complex and sticky, so you really want to have a firm idea of exactly how it will work. There are a lot of little details you need to hammer out.

I'd say your first step would be to sit down for a bit and play some of your favorite RTS games, the ones that serve as your main inspiration. This time when you play, though, don't focus on zerging noobs or screaming race/gender/sexual-orientation based insults into your headset as you grind your enemy's base. (Not saying you do that, of course. Just... you know.) Instead, focus on how the UI works. Take note of what goes where, what does what, write it all down and try to incorporate what you observe into your design of how your own should work. What does each button and panel do? Under what circumstances does a panel display anything? What happens when you do odd things? Try to break the UI, try to get it to behave oddly, try to do things that no rational person would do and see how it responds.

When you feel like you have a pretty good handle on how your UI should behave (be detailed, here; don't just scrawl a few generalities on a piece of paper and get to work since that's a good way to end up refactoring down the road) then start breaking it down into pieces. Try to find the most logical places to divide the design, to break it up into small, easily achievable tasks. As you work, try to consider how each piece fits into the whole.

As you work out your prototype, if your game is still not yet mature it can be beneficial to build a stub system for testing. This would be an impostor of sorts for the game system that ultimately it will be connected to, only instead of being the fully functional thing (with all its warts and possibly non-existent emptiness) it would be a fake that would respond to the inputs from the system, provide reasonable fakes of the outputs that the UI expects to obtain, and in general provide a framework for testing the UI without all of the problems of trying to build the UI on top of a system that might not yet be fully functional. This does require that you have a strong enough idea of how your game is going to be structured so that you can build a functional fake to test against. This might be considered a form of unit-testing your UI.

I recommend using a language such as Lua or Python for building this test framework and iterating on the prototype (assuming you have familiarity with one of those languages, or another language similar to them), even if the eventual expression of it is going to be in some other language such as C++. Dynamically typed, interpreted languages such as Lua offer the benefit of very rapid iteration (no re-compilation between edits) and very free-form, ad-hoc iteration upon data structures, message-passing protocols and so forth. Once you have nailed down the design and the appearance and structure of the various underpinnings, you can then "formalized" them into your language of choice. This, of course, is very much just an opinion and if you are not comfortable in any dynamically typed language then don't bother. I just personally find it easier to do it this way.

To sort of illustrate how I have broken down things in the past, I'll talk about how I built an earlier version of the minimap for Goblinson Crusoe. I haven't yet touched any of this code for the new "look and feel" of the game, and I never really iterated on it past the first version anyway, but it was still fairly functional. Here is what it looked like (again, first-iteration stuff. It lacked a frame or any kind of decoration):



The UI widget was built in this manner:

1) It maintained a Texture representing the map. This texture was generated procedurally at level creation, when the island was procedurally generated. In an RTS, of course, you would want to implement dynamic modification of the texture, hooked to events that occur during map exploration, in order to provide fog-of-war and revealing the map as it is explored. GC doesn't have this currently.

2) It maintained a list of MapMarkers. This was a data structure that indicated what faction the owning unit belonged to, what marker to use based on what type of unit it was, etc... The minimap widget provided functionality to create and remove MapMarkers upon request, but otherwise didn't have anything to do with managing them or their contents.

3) When the map was to be rendered (after drawing the game world, during the UI rendering stage) the texture was drawn first, then the list of MapMarkers was iterated and a vertex buffer generated from all of the visible markers. A material, containing the texture atlas holding all possible markers, was bound and the buffer rendered in a single pass. Each MapMarker held all the relevant information to construct the quad representing the marker: faction (to be compared against the player's faction to differentiate hostile/neutral/friendly units and assign a color to suit), marker (combat units displayed a circle, resources displayed a blue dot, home base a green dot, etc...) location(updated by the unit's controlling component) and so forth.

4) Each game unit that was to have a minimap presence was given an instance of MapMarkerComponent. This component would listen for some events pertinent to managing the map marker: position events in order to move the marker when the unit moves, spawn event to request a MapMarker from the minimap widget when the unit is first spawned, kill event to remove the MapMarker when the unit is removed, marker type, etc... This component acted as the go-between, hooking the unit up to its minimap representation. All communication lines between unit and its minimap representation occurred through this component.

Of course, in an RTS the minimap needs to be responsive to mouse clicks in order to move the game camera view to center on the clicked location. Such things would need to be carefully designed and implemented, probably through another proxy type of component assigned to the camera object, and highly depended upon your particular architecture.

Now, there are third-party libraries available that provide GUI functionality. These can be very useful when building your UIs, but be warned that most are rather generic and require quite a bit of tweaking (by way of style sheets, perhaps, or through other means) in order to adapt them to your use. Also, most are typically designed around the Button/ListView/RadioButton/etc... standard of UI construction, and not so much around the Minimap/RTS Unit Panel/RTS Building Construction Panel school of thought, so you might need to get your hands dirty in the guts of the library either tweaking existing widgets to suit your needs or, possibly, deriving new widget subclasses to do what you require. Still, it can be highly instructional to look at one of these libraries and work with it to get a feel for how UIs are built and how they operate.

UIs are tricky. They're pretty much my least favorite thing to fiddle around with. They are a necessary part of the process, though, and since the player is going to spend every single minute of play time interacting with your game by way of the UI, it is vital that you get it right, and that will only come through a lot of hard and (imo) tedious work.

Good luck.

Edit: One other thing. You've heard the term "Make games, not engines"? The same applies for UIs. If you roll your own UI system, you might be tempted to go crazy and start implementing everything that other UI systems implement. Resist. If you find yourself writing code for a scrollbar, then it better be because your design requires a scrollbar. It can be very easy to fall into the trap of writing a general-purpose UI library when you just don't need one, and that can eat up a lot of time.

JTippetts: It's really challenging for me to use software design, the way I've been doing it so far is just coding without design at all a little bit every day. The result is that it grows sort of amorphously and I end up going back and tweaking a lot of stuff. Even though I use C++, I don't use object oriented design because I think to myself "linux is written in c" and so I have used that as an excuse in the past. This UI will be a good opportunity for me to design it clearly and then implement it based on a blueprint. So from your reply where you talk about "proxy type of component for the camera" you mean that all the buttons on the UI have an object oriented class object (or as I would have used in the past, a struct) that the buttons talk to when they are clicked and then the proxy objects talk to whatever unit receives the command. I'll design a little bit every day for a while before I start coding the UI.

You might want to read up on on some design patterns, and make a serious effort to learn how to engineer a system. Just because Linux is in C rather than C++ doesn't mean there isn't a hefty chunk of design going on.

You can iterate on design, that is fine. I myself have never worked very well doing all of my design work up front, then trying to code the system perfectly from that design. That approach doesn't work for very many people, actually. However, there does need to be design done as you work, and you really, really need to engineer a sensible framework or system upon which everything will be built. Or else, as you put it, everything grows "sort of amorphously". Try working with that amorphous codebase three years down the road, when you need to fix a newly discovered game-breaking bug, only the code that you wrote that contains the bug is a huge steaming pile of spaghetti with dependencies that thread everywhere in such a brittle fashion that the slightest of changes has ripple effects that propagate throughout the entire project. It can happen, but you can choose your design so that it does not.

In Goblinson Crusoe, the framework I chose is a component system. (You can read the article section here for some good introduction to component systems.) Basically, each object is a generic container for a collection of components that define the behavior and characteristics of the object. Is an object a combat unit? It needs a Vitals component (to hold life, food, elemental powers and resistances, etc...), it needs a CommandQueue component (the puppet master that executes low-level tasks such as take a step, cast a spell etc...) a FloatingCombatText component (listens for events regarding combat, such as damage, healing, status changes, etc... and queues a text message up to be displayed above the unit's head) a BuffsDebuffs component (container for status-modifying buffs and debuffs) the various components for animation and hosting of particle effects (AnimatedModel, AnimationController, AnimationMap, etc...) and a few others. Each combat unit also needs a PlayerTag component to define faction affiliation, or who owns what, as well as a Controller (Human Player or AI controller variants exist) and any auxiliaries that are required. Human Player units possess an additional set of components acting as go-betweens for the UI: a PortraitTabSelectionProxy acts as a go-between for the unit and it's selection portrait, which is the portrait the player can click on to select that unit during the player's turn; when the frame is selected via the mouse, a Selected event is passed from the UI widget to its owning object via the proxy component, telling it that it is now active and ready to be controlled by the player. (GC is a turn-based RPG, by the way.) The player unit also has a QuickBarProxy that acts as a go-between for the player's row of quickbar buttons and skills. When a hotkey corresponding to an action button is pressed, or when a button is clicked, the button generates an action event that is passed, via the proxy, to the unit. And so on, and so on.

Note the use of proxy components in this case. The UI widget gathers some input and decides, based upon the input, that something needs to be done to a particular object. But since objects can be comprised of pretty much any combination of components, and since objects are generic containers rather than some kind of interface class, there is just no built-in way to tell the object what it needs to be told. The proxy is the UI widget's "way in" to the unit, the mouthpiece by which it can speak to the object, using the particular "language" that is central to the widget's behavior and which the proxy understands. The proxy is responsible for handing that communication on to the object. Since the UI widget talks to the proxy rather than, say, directly to the Player Controller you are free to build different Player Controllers (perhaps for different kinds of control schemes or types of units) without having the code the widget to talk to all of the varieties. You limit the dependency of a player-controlled unit upon the UI code to one specific module (the proxy) rather than spreading that dependency out among various classes.

All of the complex behavior of a combat unit, then, is broken down into these kinds of modular "pieces" that are plugged in to the object to build complex behaviors. Each nugget has been broken down into specialized tasks, and each works in relative isolation without having to be aware of the other pieces, although there are, of course, a few places where dependencies are built-in. An AnimationController, for example, is useless without an AnimatedModel to work on and does its work directly upon the AnimatedModel if it exists. Similarly, the AnimationMap (which converts events such as Start Walking, Stop, Turn and Melee Attack into Play Animation commands passed to AnimationController) depends upon AnimationController. But other components do what they do in mostly complete isolation. FloatingCombatText, for example. It doesn't have any explicit dependencies to any of the other components, nor do any of the other components have any explicit dependency upon it. Its sole purpose is to eavesdrop on the events being passed (the primary means of communicating with an object is through events), cherry-pick the ones it is interested in, and build a floating text message in response.

The basic framework I am now using for this system is provided by the Urho3D library, which builds the foundation of the component and messaging systems as well as rendering abstraction and so forth. That is what I mean by choosing a framework. Either you pick a library/engine that has such a framework built-in, or you design your own custom-built framework from the standpoint of creating a flexible enough system that you can realize your design upon it and it will help you to constrain your design to something that hopefully won't end up as a pile of spaghetti with weird tendrils all over the place.