I haven't gone over the code in your link (sorry) but the problem you described sounds like a typical problem with simple collision response. When you detect a collision, are you pushing the objects apart before applying a collision response force? If not what will happen is in the next frame the object will try to move apart but are still overlapping/colliding and a collision response force will be created in the opposite direction. The result of that is objects that vibrate and/or explode very quickly.

Ideally you'd be doing collision detection that takes an objects velocity into account over a particular frame. So in one frame, you would detect when an impact will occur, move the object to the point of collision, apply collision response, then with any time remaining in that frame you move the object by its final velocity (which would be away from the point of impact).

The method I described above works well in simple scenes, but breaks down when many objects are colliding at once and the math becomes fairly complicated.

I'd do it like this:

void GetProjectionOnAxis(const Vector2 * pVertices, const int N, const Vector2& axisN, float &projMin, float &projMax) {
    projMin = projMax = Dot(pVertices[0], axisN); // axisN assumed to be normalized
    for (int i = 1; i < N; ++i) {
        projMin = Min(projMin, pVertices[i]);
        projMax = Max(projMax, pVertices[i]);
    }
} 

float GetOverlapOnAxis(const float projMinA, const float projMaxA, const float projMinB, const float projMaxB) {
    float minP = Min(projMinA, projMinB);
    float maxP = Max(projMaxA, projMaxB);
    float overlap = maxP - minP;    // negative if no intersection
    return overlap;
}

struct SATResult {
    Vector2 m_normal;
    float m_depth;
    bool m_intersect;
};

bool TestPolygonPolygonSAT(const Vector2 * pVerts0, const int n0, const Vector2 * pVerts1, const int n1, SATResult& satOut) {
    jlVector2 edge, edgePerp, edgePerpN;    
    float min0, max0, min1, max1;
    float overlap, minOverlap = -F32_MAX;
    jlVector4 minNormal;
    satOut.m_intersect = false; 
    int j, i;
    for (j = n0 - 1, i = 0; i < n0; j = i, ++i) {
        // compute our orthogonal axis for each edge
        edge = pVerts0[i] - pVerts0[j];
        edgePerp = RightPerp(edge); // y, -x
        edgePerpN = Normalize(edgePerp);
        // get min/max projection on each axis
        GetProjectionOnAxis(verts0, n0, edgePerpN, min0, max0);
        GetProjectionOnAxis(verts1, n1, edgePerpN, min1, max1);
        // if no overlap, no intersection
        overlap = jlOverlapOnAxis(min0, max0, min1, max1);
        // otherwise store minimum overlap value and the normal
        if (overlap < 0.0f) {
            return false; // have to intersect on every tested axis for SAT to return true
        } else if (overlap < minOverlap) {
            minOverlap = overlap;
            minNormal = edgePerpN;
        }
    }
    // now do the same thing, on pVerts1 axis
    for (j = n0 - 1, i = 0; i < n0; j = i, ++i) {
        // compute our orthogonal axis for each edge
        edge = pVerts1[i] - pVerts1[j];
        edgePerp = RightPerp(edge); // y, -x
        edgePerpN = Normalize(edgePerp);
        // get min/max projection on each axis
        GetProjectionOnAxis(verts0, n0, edgePerpN, min0, max0);
        GetProjectionOnAxis(verts1, n1, edgePerpN, min1, max1);
        // if no overlap, no intersection
        overlap = jlOverlapOnAxis(min0, max0, min1, max1);
        // otherwise store minimum overlap value and the normal
        if (overlap < 0.0f) {
            return false; // have to intersect on every tested axis for SAT to return true
        } else if (overlap < minOverlap) {
            minOverlap = overlap;
            minNormal = edgePerpN;
        }
    }
    // if we get here they are intersecting
    satOut.m_normal = minNormal;

    satOut.m_depth = minOverlap;
    satOut.m_intersect = true;
    return true;
}

 

You would probably want to break the TestPolygonPolygonSAT into a helper function you call twice. Also, I was lazy and just did this on the spot. Pretty sure it is right though .

I would test the intersection code in isolation. Debug draw the returned normal and depth and if they are intersecting. If that seems accurate, it is probably an issue with the response code. Also, post some more of the response code here. What you have there isn't really telling us much. Got to see the nitty gritty math to help determine what's wrong.