I've read a bit about them and it seems like they hold 4 values only - an axis and an angle.

Yes: Quaternions are defined by 4 scalars. No: They don't hold an axis and an angle. There is another rotation representation that is called "axis/angle representation" and that actually hold, well, an axis and an angle. IMHO it is best not to try to interpret some geometric meaning into quaternions; look at them as being a mathematical construct in your toolbox.

In fact, each representation has some degree of freedom: In 3D, a rotation matrix has 9, a quaternion has 4, an axis/angle has 4. Because rotation only has 3, you need to enforce some constraints on the representations, or else your representation doesn't mean a pure rotation.

In the case of rotation matrices the column/row vectors are pairwise orthogonal and each one is a unit vector. Hence the process of "re-ortho-normalization", which carries both orthogonalization as well as normalization in its name.

In the case of quaternions they need to be of unit length. Hence, to be precise, one has to talk about "unit-quaternions" and not just quaternions when referring to pure rotation.

Because of this, quaternions are not less prone to numerical errors per se, but they are more stable in that numerical imprecision has less effect on still representing a useable rotation. So re-normalisating quaternions needs to be done less often, and ensuring that single constraint is less work than ensuring the 6 constraints on rotation matrices.

That said, using quaternions and re-normalization (from time to time) and on-the-fly to-matrix conversion would be sufficient for sure.

What I don't understand is how I can use a quaternion to hold the entire orientation of my object?

The problem of lacking a geometrical interpretation suitable for humans, doesn't mean that unit-quaternions don't work. A unit-quaternions "holds the entire orientation" by its nature. Use conversions from or into, resp., other rotation representations where meaningful.