I dont think you have much of a choice. Dealing with non convex geometry will be very hard to pull off. Detection isn't the problem, but it is to get a fluid, continuous response.

Well I wouldn't say I'm a pro exactly but I do have convex hull versus poly soup (my world) collision working in my physics engine.

I actually use a swept SAT collision tests, but it could probably be adapted to use GJK or whatever you want.

Basically I simply perform an SAT test between my convex hull and each world triangle individually (broad phase culls out most of the world triangles so there's only a few to perform SAT tests on).

In order to get correct collision response, I pre-process my poly soup collision so that each triangle has a flag for each edge. This flag determines if an object can actually collide with the edge - it is only true for edges that face outwards (this can be easily determined by using the dot product of the 2 triangles that share the edge).

eg. think about the edges of a simple staircase - the external edges of each step can be collided with, but the internal edges between each stair cannot because you would hit the connected faces before the edge.

e = external edge
i = internal edge

_*e
*|_*e
i**|_*e
**i**|_
****i**|

(this is a side view, where e and i are edges that go into the screen of the 3d world - I hope my crappy ascii pic makes sense - I had to replace the spaces with * to keep the formatting)



Then all I did was modify the SAT test so that all the (convex hull edges cross triangle edges) are still used as a separating axis test, but the MTD (minimum translation distance) is only updated if the triangle edge is marked as collidable.

For swept sphere collision, I also extended the system so that each triangle has an additional 3 flags (1 for each vertex) to determine if a vertex should be considered during swept sphere versus triangle tests.

The same idea is also discussed by the novadex guys in Game-Programming-Gems4.

Let me know if this doesn't make sense or you have further questions,
Steve.

[Edited by - sbroumley on November 9, 2005 8:34:38 PM]

along similar lines, in the first commercial game I worked on, I computed smoothed normals at each of the vertices. When a collision with a vertex occured, I took the normal at the vertex, and not the MTD, for calculating the colliison response.

If it was an edge collision, I interpolated the normals at the edge vertices to where the contact point was found, and used that as my collision normal.

It's a bit nasty, but the problem is nasty. Even with swept SAT, and flagging edges-vertices, you can get unwanted 'hard' collisions, and smoothing them out will be a pain.

problems :

- the collision mesh has to be reasonably tessalated. Long edges will produce errors.
- need to store normals for vertices.
- Other problems I forgot about! (I's been 5 years :)).