Just like the others have said above, you'll have a hard time finding a book for 3D graphics mathematics in Java. Also, as others have stated, LWJGL is a good place to start if you want to access OpenGL using Java -- this is what I'm using, and just because I'm more familiar with Java, I can get better performance with that language than with C++ (note I said with my experience; C++ will always be faster, its just that I know Java and how it works much, much better and can plan for that and I can program better using Java).
Also, there's the JMonkeyEngine Source Repository that you can always look at for examples (src is in the trunk folder). JMonkeyEngine is the source for an engine written in Java using OpenGL, and you can check out how they write their algorithms for mathematical computations (it's how I learned quaternions). It will take a little more work on your part (you need to navigate the source code) but you can learn sooooo so much from reading the source -- and it's in Java!! Check it out if you feel like it'll help