GPU does the math too. I would say it's more like, CPU does the logic that brings everything together, and GPU is the powerhouse that drives rendering, and possibly physics (and perhaps other stuff too).
I think about it like this:
CPU does the math, GPU does the rendering.
Ta-Da (I'm not a smart programmer ).
Perhaps the best way to say it is that a CPU is a generalized processor, and a GPU is a processor designed to efficiently operate on graphical entities. It's more or less a very specialized CPU - what it is designed to do, it does much faster than a general CPU, and what it wasn't designed to do, it does much, much, much slower.