This is nontrivial to say the best, you might want to look at gimpact, I've been told it works fine with arbitrary meshes. Or at Bullet, which integrates gimpact as well as far as I've understood.
I've tried to investigate the gjk method but this only works for convex shapes and my walls are too complex to be reduced to convex pieces (they may be long and very twisty, with a complex local surface), so a triangle-triangle based algorithm would be preferable; anyone has any idea on how to solve this problem?
Also consider separate collision geometry. Graphic meshes build up in complexity much faster than collision can handle.