I worked as physics programmer my whole career. Initially I implemented a physics engine for Lair - on the PS3. After this I worked at Havok in the destruction and physics team. After some time as freelancer where I most mentionable implemented a bunch of subsystems (contact creation and solving) for Ubisofts inhouse Motion engine which is now used in e.g. Ghost Recon. Lately I integrated Blizzard inhouse engine used in Diablo 3 into Starcraft 2 - Heart of the Swarm. Now I am at Valve and do physics again
From my experience there are two things. Physics engine development and physics engine integration. Usually people are somewhat familiar with one the two though I argue it definitely helps to understand both ends. Obviously understanding the limitations of physics engines helps with integrating them. On the other end understanding the common requirements of a game engine helps with implementing the needed features in a physic engine.
For writing physics engines there is indeed not much bundled information out there. There is lot of information and there is a lot of useless information. I think in the first years I spend a lot of time reading through useless crap, picking up a gem every here and there. If you want to learn about physics engines I recommend taking Box2D Lite and port it to 3D. Then iterate from there with broadphases, block solvers, different friction, contact models or whatever interest you. One thing that is is really important is that the requirements for physics engines are changing. From my experience constraint solvers and ragdolls are nice, but what is much more important is is fast collision detection. This includes containment queries and any kind of casting rays, boxes, capsules, etc. So usually when you become a physics programmer you also become an expert on collision detection. Another thing that is always forgotten (even in some commercial packages until more recent times) is geometry processing. E,g convex hulls, convex hull simplification or convex decomposition. A solid understanding of computational geometry is very helpful.
For the integration part this is usually setting up the game engine to route collision queries through the physics system and attach it to the animation system. E.g. switching to ragdolls from animations and inherit the velocity. Or drive to some pose from where ta character can stand up. So on this end a good idea of a general game engine workings and of course the animation system in particular is very helpful.
Regarding your question about graphics programming I have a solid understanding of OpenGL and DirectX, but I would not be able to write a deferred renderer without some upfront reading into this topic. And to be honest this was never required. Usually you setup a simple rendering for your testbed for your physics engine which is quite trivial or you setup the current render system in the game to do your debug rendering. Personally I found that the better graphics programmers always a some artistic understanding which I am missing totally.
So if you want to learn physics programmer I recommend looking into Box2D and all of Erin Catto's presentations. Get something up and running and then experiment with this. A good start is a simple pendulum and a falling sphere. Then a box, a stack, etc. Don't try to reinvent everything, but make sure you understand the current state of the art first including its limitations.
Besides Erin's work you can also look for publication of Claude Lacoursiere. He was one of the initial engineers on the ODE while it was developed at MathEngine and he has put some good lectures out there at Umea (a swedisch university). You can google for "Spook" as well in the context.
For collision detection I recommend the books from Gino v.d. Bergen and Christer Ericson.
I hope this helps a bit. Good luck!
-Dirk
