FWIW

I created my own physics engine that ensures that all motion is constant-velocity and all changes to velocity happen in instantaneous "key frames" (impulses?) that are created at least N times per second but also for every collision

this allows me to scrub backwards and forwards in time by using simple linear interpolation between key frames

also it means that I swap some simulation accuracy (mainly drag effects) for "perfect" collisions so I don't have any interpenetration issues

(so far I haven't needed more than 1000 frames with up to 16 objects (pos + quaternion) covering around 12 seconds)