Well if you want to put the game up on Xbox Live then you must use 4.0. I believe it's been rearchitectured to take advantage of DX 10 and 11. Otherwise it doesn't make a difference.

You are correct. Though I remember reading somewhere that they refactored the code due to DX 10/11.

The direction XNA went with 4.0 can be a touchy subject. If you want to get involved with their live arcade or windows phone stuff, it's the only way to go (3.1 games won't be accepted). It also has (in my opinion) a cleaner/nicer graphics API (also the API is split up between several assemblies now) because its organization is based on the organization D3D10 started (and D3D11 expanded). However, it still is D3D9 under the hood. This has drawn some criticism because the HiDef profile requires you to have D3D10 level hardware, but doesn't expose the newer D3D10 features.

Also, you do lose some freedom in 4.0 when it comes to loading effects/textures at load time (the static load from/compile from files are gone). This means you have to either roll your own runtime importer or use the content pipeline (which has a different license/not a part of the redistributable, and is Windows only - of course those static methods were always windows only anyways, which is why they were removed). However, for effects, you're stuck using the content pipeline since the compiler they use is supposed to be slightly tweaked from the standard FX compiler. They also removed the Vertex/Pixel shader objects and everything's an effect, in 3.1 you could potentially put together shader fragments but you can't in 4.0.

For the majority of users these may never pose a problem, and personally I think the enhancements do outweigh the negatives. The unfortunate problem with 4.0 is it kind of felt like Windows was a "nice to have option" and MS was more interested in xbox 360 and windows phone support. But the benefit of learning the XNA 4.0 graphics API is if you do feel you want to break out and make a change (e.g. going to SlimDX, or even native DirectX), then you'll already be somewhat familiar with the D3D10/11 API organization.