In my opinion it is not. It wouldn't be a bad thing, but most of what is in those books you don't need.

However........there are some things you will likely need to understand. For example, though you don't need to know how to calculate a dot product, you need to know what it is used for, as in what problems you need it to solve, as in where to use the thing. It also applies to quaternions. Example, if you have an asteroid, and you shoot it with a bullet. The bullet comes from the lower left, so you want to create asteroids exploding off from the original large asteroid. You'd need to know to use a vector subtraction to figure out from what direction came the bullet(or where it hit). Then you'd need to rotate it various degree amounts(+-30 for example) to figure out where to create the new asteroids.

Another thought, if you are wanting to create your own shaders, some of the matrix knowledge will be needed to an extent in order to calculate things that don't have direct function calls for in shader languages.

Bottom line, you don't need to know the gritty details, but a somewhat basic geometry overview should get you started, followed by a sort of primer to how these things work, without having to memorize the underlying formulas.