For 3D math related to games I would recommend:
3D Math Primer for Graphics and Game Development, Second Edition.
The book is not dumbed down but presents the math as it relates to graphics and games.
I actually have that sitting right next to me. I've only gotten a few chapters in and I do a little bit of it here and there, but I'm wondering if I can continue with Introduction to 3D Game Programming with DirectX 11 without really comprehending the entirety of it or if I should really wait until my math is there. I'm fairly impatient and have been longing to get acquainted with DirectX for a long time... and in general I feel that I'm very competent with math, I've gone through calculus and discrete math previously (although I'm redoing it again soon).