I really liked my GameBoy Advance SP (the SP version of the GBA) - I think it's form factor was very nice.
Backlighting of the screen, flip-up "laptop-like" form with a heavy duty hinge. Compact, amazing graphics, great games.
I have large hands, so my hands were slightly cramped - I suppose the wider Nintendo DS improved on it by being wider, but I never owned a DS.
The Dreamcast was awesome, but without many games, and I think the PlayStation 2 really improved and refined the first PlayStation.
I started with the N64, so my experience doesn't go any farther back.