I'm working on a small rigid body simulator, and this has been a long haunting problem for me: how do I get penetration normal from a bvh. Eventually, by traversing the structure, we can get all pairs of intersecting triangles, however we still don't know if each of the six vertices (from each pairs of triangles) are inside or outside the object itself, nor the normal to the object surface.
I thought I can solve this problem by using a signed distance field, where the inside test is straightforward and the normal can be acquired from discrete differentiation with near by cells, however, when constructing the distance field, I still need to know if a point is inside the object or not. I used to assume the object to be convex (and the problem is easy to solve), and when I move to concave objects, it seems to be very difficult. The ray shooting method is slow and not robust enough, what's worse is that I got some models that are not simple polyhedrons (meaning that the object has multiple components and there are welding edges between them, so that some triangles might be inside the object!)
So after all, my questions is: how to decide if a point is inside/outside the object with BVH, assuming the polyhedrons are normal (those irregular ones can be decomposed)? I know BSP can do this, however I believe BVH can do it as well, otherwise there won't be SO much literature about BVH-based collision detection, the weird thing is that almost nobody mentioned this problem among the literature I searched, am I just being paranoid to solve a irrelevant problem? The only reference I found about this problem is "A Fast Method for Local Penetration Depth Computation" from M.C. Lin, which is somewhat over complicated to me. A quick distance field construction reference is welcome as well.
Or are there other ways to pass around the problem and get the collision normals directly from BVH?