I'd say your first step would be to sit down for a bit and play some of your favorite RTS games, the ones that serve as your main inspiration. This time when you play, though, don't focus on zerging noobs or screaming race/gender/sexual-orientation based insults into your headset as you grind your enemy's base. (Not saying you do that, of course. Just... you know.) Instead, focus on how the UI works. Take note of what goes where, what does what, write it all down and try to incorporate what you observe into your design of how your own should work. What does each button and panel do? Under what circumstances does a panel display anything? What happens when you do odd things? Try to break the UI, try to get it to behave oddly, try to do things that no rational person would do and see how it responds.
When you feel like you have a pretty good handle on how your UI should behave (be detailed, here; don't just scrawl a few generalities on a piece of paper and get to work since that's a good way to end up refactoring down the road) then start breaking it down into pieces. Try to find the most logical places to divide the design, to break it up into small, easily achievable tasks. As you work, try to consider how each piece fits into the whole.
As you work out your prototype, if your game is still not yet mature it can be beneficial to build a stub system for testing. This would be an impostor of sorts for the game system that ultimately it will be connected to, only instead of being the fully functional thing (with all its warts and possibly non-existent emptiness) it would be a fake that would respond to the inputs from the system, provide reasonable fakes of the outputs that the UI expects to obtain, and in general provide a framework for testing the UI without all of the problems of trying to build the UI on top of a system that might not yet be fully functional. This does require that you have a strong enough idea of how your game is going to be structured so that you can build a functional fake to test against. This might be considered a form of unit-testing your UI.
I recommend using a language such as Lua or Python for building this test framework and iterating on the prototype (assuming you have familiarity with one of those languages, or another language similar to them), even if the eventual expression of it is going to be in some other language such as C++. Dynamically typed, interpreted languages such as Lua offer the benefit of very rapid iteration (no re-compilation between edits) and very free-form, ad-hoc iteration upon data structures, message-passing protocols and so forth. Once you have nailed down the design and the appearance and structure of the various underpinnings, you can then "formalized" them into your language of choice. This, of course, is very much just an opinion and if you are not comfortable in any dynamically typed language then don't bother. I just personally find it easier to do it this way.
To sort of illustrate how I have broken down things in the past, I'll talk about how I built an earlier version of the minimap for Goblinson Crusoe. I haven't yet touched any of this code for the new "look and feel" of the game, and I never really iterated on it past the first version anyway, but it was still fairly functional. Here is what it looked like (again, first-iteration stuff. It lacked a frame or any kind of decoration):
The UI widget was built in this manner:
1) It maintained a Texture representing the map. This texture was generated procedurally at level creation, when the island was procedurally generated. In an RTS, of course, you would want to implement dynamic modification of the texture, hooked to events that occur during map exploration, in order to provide fog-of-war and revealing the map as it is explored. GC doesn't have this currently.
2) It maintained a list of MapMarkers. This was a data structure that indicated what faction the owning unit belonged to, what marker to use based on what type of unit it was, etc... The minimap widget provided functionality to create and remove MapMarkers upon request, but otherwise didn't have anything to do with managing them or their contents.
3) When the map was to be rendered (after drawing the game world, during the UI rendering stage) the texture was drawn first, then the list of MapMarkers was iterated and a vertex buffer generated from all of the visible markers. A material, containing the texture atlas holding all possible markers, was bound and the buffer rendered in a single pass. Each MapMarker held all the relevant information to construct the quad representing the marker: faction (to be compared against the player's faction to differentiate hostile/neutral/friendly units and assign a color to suit), marker (combat units displayed a circle, resources displayed a blue dot, home base a green dot, etc...) location(updated by the unit's controlling component) and so forth.
4) Each game unit that was to have a minimap presence was given an instance of MapMarkerComponent. This component would listen for some events pertinent to managing the map marker: position events in order to move the marker when the unit moves, spawn event to request a MapMarker from the minimap widget when the unit is first spawned, kill event to remove the MapMarker when the unit is removed, marker type, etc... This component acted as the go-between, hooking the unit up to its minimap representation. All communication lines between unit and its minimap representation occurred through this component.
Of course, in an RTS the minimap needs to be responsive to mouse clicks in order to move the game camera view to center on the clicked location. Such things would need to be carefully designed and implemented, probably through another proxy type of component assigned to the camera object, and highly depended upon your particular architecture.
Now, there are third-party libraries available that provide GUI functionality. These can be very useful when building your UIs, but be warned that most are rather generic and require quite a bit of tweaking (by way of style sheets, perhaps, or through other means) in order to adapt them to your use. Also, most are typically designed around the Button/ListView/RadioButton/etc... standard of UI construction, and not so much around the Minimap/RTS Unit Panel/RTS Building Construction Panel school of thought, so you might need to get your hands dirty in the guts of the library either tweaking existing widgets to suit your needs or, possibly, deriving new widget subclasses to do what you require. Still, it can be highly instructional to look at one of these libraries and work with it to get a feel for how UIs are built and how they operate.
UIs are tricky. They're pretty much my least favorite thing to fiddle around with. They are a necessary part of the process, though, and since the player is going to spend every single minute of play time interacting with your game by way of the UI, it is vital that you get it right, and that will only come through a lot of hard and (imo) tedious work.
Edit: One other thing. You've heard the term "Make games, not engines"? The same applies for UIs. If you roll your own UI system, you might be tempted to go crazy and start implementing everything that other UI systems implement. Resist. If you find yourself writing code for a scrollbar, then it better be because your design requires a scrollbar. It can be very
easy to fall into the trap of writing a general-purpose UI library when you just don't need one, and that can eat up a lot of time.
Edited by JTippetts, 25 April 2013 - 09:19 PM.
Sit down and work out a design. UI code can quickly become complex and sticky, so you really want to have a firm idea of exactly how it will work. There are a lot of little details you need to hammer out.