I have done quite a few modifications to the engine to support what I'm trying to do with it. Is there a common cause of this that I can look for?
Bugs in your engine modifications, unless the original Jitter engine has the same problems.
How can you expect help without giving any detail of what you are doing? Common immediate causes of "flying off in random directions" include forces proportional to interpenetration depth (of unexpectedly large magnitude) and naive choices for the direction of such forces, but your problem might be less common.
In any case, your choices of accuracy tradeoffs (e.g. allowing interpenetration in order to limit forces), collision response rules (e.g. constraining bounce directions) and physically meaningful hacks (e.g. inelastic collisions) depend on your game's needs.