Can you be more specific when you say "non-rectangular"?

Also, could you be more specific as to if you mean the main window or just the GUI elements inside?

It should be pretty straight forward to do. With rectangular widgets you just test to see if the cursor position is inside the rectagular bounds of the widget. In the case of "non-rectangular" widgets, you do one additional test to make sure the pixel under the cursor position is a non-transparent color. So:

if(widget.containsPoint(cursor.x, cursor.y) && widget.getPixel(cursor.x, cursor.y)!=widget.getTransparentColor())
{
widget.fireMouseOverEvent();
}

SDL does not provide any "getpixel" or "setpixel" functions however you can use the SDL documentation to write your own. Once you have the pixel value you can use SDL_GetRGBA to get the individual color channels from it. For example this is taken from the SDL documentation which will answer most of your questions:

A better idea if you want to think in terms of "windows" is to actually use rectangles.

The green rectangle is where you start checking for mouse events in that particular window since that is the window holding all of the "controls". The red rectangles are the ones that actually do anything. If the initial check for the mouse position inside the green rectangle returns false then you don't check any further. If it returns true you check the control rectangles and act accordingly. Checking anything else is wasted in this case.

A simple parent->child relationship is key to making this work efficiently. Point in rectangle checking is relatively cheap but partitioning space in this way helps you eliminate unnecessary checks making it even more efficient.