I solved the problem by using the smooth vertex normals as the plane of collision. So, for an edge-AABBox collision, instead of having the edge hitting the box dead-on and catch the edge, I linearly interpolate the two vertex normals of the edge, find the normal at the point of collision, and use that to make the plane of collision.

Alternatively, you can use an algo similar to a phong-shading, and linearly interpolate the normal all the way accross the triangle. That should give you a smooth surface, but maybe too smooth. Just using that on the edges did the trick for me. And obviously, it counts also for the triangle corners.

                                   ^              \              ncoll |        /               \                   |       /                \                  |      /                 +-----------------*-----+                                 pcoll

If you use dynamic collison detection (using velocity or displacement between frames), this should also help you get a good sliding effect.

http://www.gamedev.net/community/forums/topic.asp?topic_id=166540



a note, to get the point of collision on the box, I had to ''raytrace'' inside the box, along the normal of collision down te normal of collision.

And for your information, Quake uses a BSP tree, which takes away the pain of dealing with edges and corners. You might want to investigate that. Look at Stan Melax and ''Dynamic Plane Shifting'' on Google.