Honestly, the most important math you could know for a solid foundation to build off of would be trigonometry and linear algebra. Ultimately you're working with planes (triangles) aligned within a 3D space (2D if you're making a 2D game). Knowing your trig will be of huge benefit when it comes to triangle manipulation.

For super basic DX and starting to learn you really only need matrices and vectors. That's what they taught us at uni and then when you start getting more advanced you're looking at trig, pythag and others similar to that.

To use directx, you don't need to know any math. To do anything useful with directx, you will need to know math (as covered already by other posters in this thread). I started off in 3d graphics without knowing any math. You can only get so far, and even then it's a frustrating and precarious road as it's like trying to understand complex concepts in a foreign language you have little grasp on: it's just another layer of complexity that makes a hard task even harder. Luckily, as Bacterius highlighted, the bare minimum is pretty self contained, i.e. you can pick up a math book with a gaming slant and start making headway. From my personal experience, I cannot emphasise this enough: if you're interested in graphics, learn the math! If you don't want to learn the math because you find it boring and lack the motivation then you can pretty much drop your ideas of working with low level APIs like GL and DX or any low-level graphical work. If your primary motivation is making games then off the shelf engines would be a far better option, although the bare minimum (cartesian coordinate system, vectors and, to a slightly lesser degree, matrices) is still important (and, in all fairness, pretty easy to grasp, even with a phobia of math).

Its been my dream to make a crappy hobbyist quality 3d engine, but if i need to know a truckload of math that would be a hearbreaker =(

Bear in mind that a lot of the knowledge you need to do this to the level you mention is actually what you can pick up from DirectX tutorials and books, which you'll have to do anyway if you're new to DirectX.

A couple of people have commented on that sort of thing as a possibility and/or whether or not it's a good idea, but I would classify it as giving it as go or 'making headway' as GeneralQuery mentioned. I think the advantage of doing that is that you'll learn more about your own limitations during the process, including understanding whether you need to pick up a little more math knowledge.

If you already know calc 1 then do consider that through practice and application, the fuzzyness you describe will disappear and you are somewhat already set up to be filling in blanks from other math topics.

Also bear in mind that there are many 'graphics programmers' out there who don't necessarily operate at the cutting edge of R&D and are quite often following others. Really, the idea these same people are top math students or not is more a measurement of how much more easily they'll be able to devour white papers to follow new research but even the lack of an ability to do that does not preclude a lot of people from developing 3D engines. For many, it just means they're devouring knowledge at a different (higher) level, perhaps after others have turned said white papers into tutorials and pseudo code, in other cases it's often using libraries developed by others and gradually picking up the knowledge that is initially quite alien.

You don't necessarily need to know the implementation details or derivation of half of the formulas, but you will need to understand conceptually how to use the math libraries, and how to use them to compute the proper transforms.

If you do only 2D, the math involved is simpler, but not necessarily by much. It depends a lot on what you are doing.

Here are some links to test yourself:
- Do you think you can understand a math library like this and how to utilize it e.g. to specify object positions, move them around, rotate and scale them?
- Are you familiar with the Direct3D pipeline stages? A lot of the math you feed into the device revolves around that architecture.
- Does the chain of linear spaces "local -> world -> view -> clip -> screen" sound familiar, and can you understand the concepts related to this?

If you feel comfortable with these, you'll be pretty well set math-wise on developing your own 3D engine. Math-wise there's not much else than basic calculus and linear algebra involved.