It is quite possible (i've seen it done on two projects) to get the notation wrong but get the matrix calcs correct and essentially cancel-out errors. i.e. When you look at the math with the operator overloading you think columns, but under hood it is doing rows. I'm not up much on my quaternion derivations, but i would guess the transpose is not correct because Matrix multiplies are not commutive.
Being clear on if you are using column or row vectors as a convention is super important though.
e.g. Break down some operations - if we have some object and then want to rotate object around y, move x+10
VecNew = VecCurrent * Mat(Translate to origin) * Mat(rotate y) * Mat(Translate to old pos + 10 more x)
As a column vector, the operation Vec*Matrix is invalid (3x1) * (3x3) is nonsense and we accumulate matricies on the left:
VecNew = Mat(Translate to old pos + 10 more x) * Mat(rotate y) * Mat(Translate to origin) * VecCurrent
Sorry if this is not what you meant and obvious.
Keep us posted though - as I am really starting to wonder what the problem is.
Suggestion: Write a few hooks to use a debug, unit sized, impulse vector oriented on x,y or z and send it into your calcs at known hit locations for each object. Ensure that this works for both cases where the normal may be inverted depending on the if it object 1 or 2.