I'm coding a Unity game, and unfortunately invested a great deal of time into it before I realized how sub-par the mesh collision system is. After plenty of time spent exhausting every available option for doing it the "Unity" way, I could still not get a platformer movement that didn't feel floaty and physics-based, while also being robust enough to handle fast object speeds. So, I set out on the ironic, degrading task of coding mesh collision from scratch in an engine I paid thousands for.
Anyway, I've gotten as far as coding intersect detection between two arbitrary meshes, using AABB octrees. Now the question I can't seem to find a robust solution for is how to find the push out vector between them. I can assume one is convex but the other (terrain) has to be non-convex. I also might be able to get away with replacing the convex mesh with a mathematical Y axis cylinder (rather than a cylinder mesh which I've been using up to this point).
Rather than fumble around, I looked for a resource on this problem which has no doubt been solved dozens of times. But I couldn't find much.
I considered buying this book but I looked inside and it seems like it doesn't have the info I need. Can anyone point me in the right direction?