After you compute a TOI you need to make sure to move the shapes 'close enough' such that the contact generation doesn't fail. E.g. the classical error is to advance the shapes to the determined TOI and when running the contact creation routine you end up an empty manifold. This will lead to the object tunneling out of the world even though you determined the correct TOI. In order to to handle this my contact creation routing considers shapes withing some margin touching and builds the manifold. I use an adjusted SAT which can handle small separations.
For me the major source of failure in the TOI computation using any form of conservative advancement was the GJK. If you run a GJK with some epsilon tolerance to support e.g. quadric shapes you run into a numerical nightmare with 32bit floating precision. So in Rubikon I only support spheres, capsules, hulls and meshes and the GJK implementation continues until it doesn't make any absolute progress getting me as close as possible. With this I haven't seen any numerical problems in practice. Note that spheres can be handled as points and capsules as segments and the radius is handled afterwards.
Erin's GJK and TOI actually address these problems, though they might not be explicitly mentioned here:
I gave a presentation a GDC on contact creating in general which might also be helpful:
Finally I recommend studying the Box2D continuous physics solution. It addresses a lot of problems with continuous physics. You will find a great example of sub-stepping there as well which I didn't discuss in my answer here.