So it turns out that the other day I was bit too optimistic - tested it with bindpose only and I presumed it's allright. What I did - I got the bind pose matrices and then inverted ones, then multiplied bind pose by inverted and then sent it to the shader (silly me, I forgot that multiplying matrix by its inverse gives identity matrix, so no wonder the bind pose object space vertices were ok). Happy to see that it works I assumed that I only need to build animated matrices and replace bind pose matrices by animated ones in multiplication. But sadly this doesn't work. The result is totally messed up. It helps a little if I invert the final matrix but it's still not right.
What I already double checked:
1. quat to matrix conversion
2. matrix by matrix multiplication
3. matrix inverse routine