So I have implemented a sequential impulses collision/contact solver including the contact manifold generation (up to 4 points for a pair of objects) in the 3-dimensional case.

Now, do I *need* a contact graph to solve for "stacking" groups of objects? Or can I just add more points to the solver without regard to where they are relative to some "root" object, i.e. just use a "flat" list of contacts.

For example if there were 3 objects and two contacts between them, that would be up to 8 contact points to solve for in the iterative solver.

To help converge faster, I am aware of "warm-starting" which is basically caching the impulses over frames for persistent contacts (of course there are issues with "identifying" contact points when the manifolds are changing). Is this all I need to worry about? No contact graph necessary?

Nope, you don't need a contact graph. You need to make sure you solve the contact points in a 'relative' consistent order over frames though. The iterative GS/PGS can be quite sensitive to varying constraint order. Usually this comes very naturally and is nothing you need to enforce. And yes, warmstarting helps a lot with stability and friction quality.