You have to do that anyway -- the C++ ABI (e.g. name mangling) isn't portable across compilers, so a library built by one compiler very likely just wont work with another compiler, no matter how simple/complex the interface is.
Another solution is to just ignore it and build different libraries for different implementations, so use msvc, gcc, clang and name your libraries after what they were compiled with. E.g. MyLibWindows64MSVC or MyLibWindows32GCC
Personally, if I can't get the source to a C++ library that I want to use, then I don't use it. Even if I can get a binary that was built for my compiler, maybe I'll want to switch compilers later and the library author will have disappeared, leaving me stuck.
But that is not an luxury that you always get.