Yea apparently using a viewport won't work since the backbuffer will still be created by default to fill the entire window. You simply cannot specify a custom buffer size. So,by using a viewport you will have all the space that is nonrendered black, and if you decide to put buttons or other controls in that space, there will be problems with rendering.

The only solution is to actually render in a control like a tab.

Not true at all.

the backbuffer will still be created by default to fill the entire window

It may be the size of the client area, but it doesn't have to fill it.

by using a viewport you will have all the space that is nonrendered black

Irrelevant. Don't present the non-rendered area.

if you decide to put buttons or other controls in that space

Just for clarity, child windows aren't put in the backbuffer. They're drawn to the client area.

EDIT: Now you've got me confused. The first rectangle in the Present parameter is set the same as the viewport rectangle. See the correction below.

Keep the backbuffer the size of the mainwindow's client area (the default) and set the viewport to the size of the rendering area desired (the size you want to appear in the client area.) For the Present call, set the first rectangle (source rect) to [0 <--EDIT: no] same rectangle as the viewport to present only the viewport area. Set the second rectangle parameter (destination rect) to the position and size in the client area where you want the scene to appear, e.g., the source rectangle with an offset relative to the client area.

Note: you can't use the DirectX presentation area in the client for anything else, but the rest of the client is yours to use.

EDIT2: If you want to use the entire backbuffer for rendering, you can also do that. Use the default viewport or set the viewport to the backbuffer size. Set the first Present parameter to 0 (present the entire backbuffer) and the second parameter as mentioned above. The entire backbuffer image will be resized (resampled, probably to your disliking) to the destination rectangle.