Ignoring row-major storage vs column-major storage for a moment, there's matrices that have been constructed to transform mathematical row-vectors and matrices that have been constructed to transform mathematical column-vectors.
Whether you're using row-vectors or column-vectors simply depends on whether you write vec * mat (mat1x4 * mat4x4), or mat * vec (mat4x4 * mat4x1). With the former type of mathematics, the forward/right/up axes will be in the first 3 rows, whereas with the latter, these axes will be in the first 3 columns (they're the mathematical transpose of each other).
Therefore, if you work with row-vectors, I'd prefer to store my matrices in row-major order, so that it's easy to extract the basis vectors (this also makes the matrix multiplication code simpler on a lot of hardware).
Likewise if you work with column-vectors, I'd use column-major storage order, for the same reason.
Personally, I work with column-vectors (and matrices designed to transform column-vectors), because that's the way that most maths material I've used teaches it -- it seems the be the prevalent convention. Therefore, I prefer column-major storage.
One place where I have to mix storage conventions is when saving space. Sometimes you want to pass matrices around as 3rows x 4columns, with a hard-coded/assumed 4th row of [0,0,0,1]. Often your registers (shader uniforms or CPU SIMD) are 4-wide, so this storage format isn't possible with column-major storage (or, it takes the same amount of space: 4 registers). So when storing a 3x4 matrix, I'd use row-major storage (which only uses 3 registers).
This same problem presents itself with the opposite convention: with row-vectors + row-major storage, you'd want to use a 4x3 matrix with an assumed 4th column of [0,0,0,1], forcing you to use column-major storage for space efficiency.
Very well put. It's often overlooked to differentiate between row vs. column-major matrices regarding operations, and row vs. column-major regarding storage.
Here's a bit more food for thought why one should try to go for column vectors and write multiplications like v' = M * v.