Yes, using an FBO provides a simple and elegant approach and is conveniently backwards compatible up to GL2.0 core. An alternative would be to draw your GUI elements on top of the 3D window every frame, or if you don't want to do that, using the stencil buffer or a discard mask when drawing the 3D scene.
For simplicity's sake, especially if you want to support all forms of shaders in your 3D scene (and you still want to do child/sibling window clipping on top of it), using an FBO is the simplest way to go.
You'd have the set up the FBO just like a regular window, set its viewport to the texture rect and just draw the scene. Don't worry about the texture copy - it's really cheap.
;) I thought so. I remember being annoyed with having to copy my texture, but if I do it right, I don't think the copy operation is actually required, although the GUI drawing would of course have to wait until after the scene is drawn, and I'll be unable to fire off all the primitives in one go. But that's a fair price to pay to be able to scale projections after rendering, besides being able to rotate everything.
I'm glad you advocate a solution allowing some backwards compatibility and easening the appearance of the overall structure. (Embedding a perspective projection inside an orthographic one seems messy, I think. )
In terms of transformations - I suggest providing the 3D scene its own transformation matrix stack and manage the GUI completely independently.
Note, however, that things get a little bit funkier if your 3D scene uses FBOs internally - just something to keep in mind.
Oh yes, about that.... Naah, I'll get around to it eventually, -But it's fun to imagine.
I picture GUIs rendered onto surfaces in space, and the space to be viewed as part of a GUI... Like ID did in Doom 3, just with no theoretical recursion maximum.