Unfortunately, you can't make a simple combination of the components as the order of application is important.

Consider: multiplying 2 matrices together, each comprised of scale-rotation-translation components. That matrix multiplication becomes the following series of operations:

scale1 * rotation1 * translation1 * scale2 * rotation2 * translation2

In your implementation, you simply add the two translations. However, the second translation (translation2) does not simply follow on translation1. Translation2 is preceded by rotation2. Rotation2 changes the orientation of the axes before translation2 is applied. So translation2 applies to a different orientation than translation1.

As a very simple example, consider the situation where an object has a translation (1, 0, 0), a move 1 unit along the x-axis. The next operation is a rotation by pi/2 about the y-axis. The next operation is a translation of (1, 0, 0). Your implementation would place the object at (2, 0, 0). However, the rotation of pi/2 changes the object's axes and the second translation actually moves the object perpendicular to its original movement to a location (1, 0, 1) or (1, 0, -1) depending on the direction of the rotation.

Your approach of maintaining separate quats and vectors for each keyframe is a good one if your intent is to lerp/slerp between keyframes, combine animations, etc. If so, you may want to take a look at my article on an animation controller for a skinned mesh. Not all of the discussion may apply to your situation, but the bottom line is that you can lerp/slerp the animation components, but you'll still have to apply them in SRT order. That is, do your lerp/slerps on the animation keyframe components; calculate a matrix combining the SRT results; and multiply the bone offset matrices by the animation matrices, etc.

It is just a matter of interpretation. Keeping the components isolated for bone animation to use slerp for the rotation is common. The interpretation of the 3 components should be: I have a bone at position X(=translation) which point in the direction of Y (=rotation) of size Z(=scale). This way you need to combine the matricies like this

// Order: first scale, then rotate, then move, applying from right to left:
final = translation * rotation * scale;

If you need the inverse, you need to reverse the order too:

// Order: first scale, then rotate, then move, applying from right to left:
inv(final) 
= inv(translation * rotation * scale) 
= inv(scale) * inv(rotation) * inv(translation)