Posted 13 October 2012 - 02:55 PM
There are a few things to consider:
1. D3DX is essentially deprecated at this point. The library is totally functional and there's nothing "wrong" with it per se, but it is no longer being developed. Any future updates will be for DirectXMath (DirectXMath is the new name for XNAMath), and not for D3DX.
2. DirectXMath is designed to map well to modern SIMD instruction sets (SSE2-4 and ARM NEON), and allows for high performance math code if used right. D3DX math can use older SSE instructions, but it does so in a sub-optimal way. One result of this is that DirectXMath has a steeper learning curve, and generally requires you to write more code since you have to explicitly load and store SIMD values. However it's possible to write a wrapper that simplifies the usage, if you don't care very much about performance.
3. DirectXMath can be used in Windows 8 Metro apps, and like I mentioned earlier supports ARM instructions. So if you ever want to release on Windows Store, you can't use D3DX at all.
With all of that said, my recommendation would be to go with DirectXMath.
Now when you talk about .fx files, what you're really talking about is whether or not you should use the Effects Framework. The Effects Framework has a bit of a weird history. It started out as part of D3DX, and was essentially a support library that helped you manage shaders and setting constants, textures and device state. Then in D3D10 it became part of the core API, and was moved out of D3DX. Then for D3D11 they moved it out of both, stripped out some functionality, and provided it as source code that you can compile if you want to use it. Once again there are a few considerations:
1. Like I said before the Effects Framework is a helper library that sits on top of D3D. It helps you manage shaders and states, but it doesn't do anything that you couldn't do yourself with plain shaders and core API's.
2. In D3D9 the Effects Framework provided a pretty good model for mapping to the shader constant setup used by SM2.0/SM3.0 shaders, as well as the render state API. For D3D10 and D3D11 it is no longer such a good fit for constant buffers and immutable state objects, at least in my opinion. Like I mentioned earlier certain functionality was stripped out for the Effects11 version, which also makes it less useful than it used to be.
3. Like D3DX you can't use it for Metro applications. This is because it uses D3DCompiler DLL to compile shaders and obtain reflection data, and this functionality isn't available to Metro apps.
Personally, I wouldn't recommend using Effects11. It's not really very convenient anymore, and I feel like you're better off just getting familiar with out how shaders, states, constant buffers, and resources work in the core API.