I find that, whenever this question is asked, invariably the answers tend to be overblown and overcomplicated. People throw around terms like "lambda calculus" and "linear algebra", but most of it is just to sound impressive and generally the programming only barely touches those concepts on a surface level, if at all.
The truth is, for most game programming (including 3D graphics), if you've gotten your 2nd year of algebra in high school, you'll be fine.
Some concepts can be expressed in multiple ways - and advanced mathematics can just give you another way of thinking about or expressing the details of a problem. For instance, Alvaro suggests complex numbers as being useful to 2D graphics - but he's really just talking about Cartesian coordinates and 2D vectors. Ultimately, in the end, the code is going to end up looking pretty similar no matter which conceptual abstraction you used in your mind to get there
The exception to this rule is if you're planning on writing a physics engine - and at that point, first-year calculus and a year of physics will generally get you all that you need.
An important, related question is - are you good at math? Because if you're really bad at math, you're probably going to have some difficulty. Not because you need the mathematics itself, but because the kind of logical thinking that goes into programming is very similar to the kind of logical thinking that goes into mathematics. So there's going to be some correlation there - if you're good at one, you'll probably be good at the other. And if you're bad at one, you'll probably be bad at the other.
But if you're good at mathematics, even if you don't know a concept important to your work, you'll be able to quickly pick up and learn what you need as you go along.