I'm trying to incorporate a sleep system into my engine. I've tried several approaches so far, and they all had significant issues. The putting objects to sleep part is simple enough, just detect when they stop moving for a certain amount of time and flag them as sleeping, and skip contacts between two sleeping objects.

The hard part is when to wake them up. Obviously, waking up any object in contact with an awake object is not going to work, because they'd just keep waking each other up indefinitely. Instead, I made them set each other's "sleep" time to the minimum of the two, so eventually the whole island goes to sleep. This works pretty well, but in some cases it's pretty useless - for instance, if I make a very long chain of dominos and start collapsing it, then the end of the collapsed chain will always keep the whole thing awake even though 90% of it isn't moving, so eventually the simulation slows down significantly.

I've checked some simpler engines and another approach I found was to only wake up objects when the awake object in the pair exceeds certain velocity. However, this means that sometimes the solver will try to solve contacts between a sleeping and awake object, so I made it treat sleeping objects as stationary (infinite mass/momentum). This fixed the domino chain, but caused a much bigger issue in other simulations. Taking the newton cradle example (or any other scene where something collides into a group of resting objects), it simply can't propagate impulses through the chain of stationary balls - if the third ball in chain never moves (since the fourth ball is blocking it), it will never wake up the fourth ball.

It seems that the waking up checks need to be repeated every time a body's velocity changes during contact resolution. This means an object has to be waken up, it's contacts with adjacent sleeping bodies need to be re-activated and initialized (which involves collision checks and some additional computations) and added to the active contact list for subsequent resolution steps.

It sounds like it could work, but I'm not sure if I like the idea of mixing the collision and resolution stages. Am I on the right track here?

Well I tried what I said, now it can never put objects to sleep, because due to gravity, at some point during the resolution step an body's speed will exceed the threshold and it will wake up every adjacent body.

Have you research Box2D's sleep mechanism? The idea is sleep an entire island of bodies by crawling an island graph, and computing the maximum linear/angular velocities. A timers is accumulated while the max are underneath a threshold (one threshold for linear, and one for angular velocity). Once the timer reaches a threshold, the island is put to sleep. Islands are awoken by force events or other user events, and if another collider enters the broadphase bounding shape for any of the island entries.

By islands here you mean any set of touching bodies? That seems like it would work similarly to my algorithm that propagates sleep times. It doesn't work well in the dominos case I mentioned - it can't put the tiles at the far end of the chain to sleep because everything is technically one island. It's also a part of the web demo I put together earlier: https://physics.rivsoft.net/Dominos - sleeping bodies can be shown with a checkbox on the right. Don't mind that I accidentally wake up extra bodies during broadphase, don't think it's the main issue here.

That is correct. Box2D's scheme will not put them to sleep.

There are a lot of potential strategies for optimization for the islands not moving much. Caching manifolds or other caching systems can be used to try and skip narrow phase.

The island building scheme can maybe be customized in some way to somehow skip non-moving bodies (like the beginning of a domino chain). But I think a caching mechanism would be better suited for this kind of problem, while leaving the island sleeping as a simple algorithm. Just my opinion.

So it looks like another problem not worth trying to solve I guess. Although I guess I could do a partial solution, where a body at least N edges away from other moving bodies (N being the number of resolution steps) is safe to leave sleeping. Otherwise I would have to wake up bodies based on the velocities in the middle of the resolution step, which doesn't seem to work well at all.

I'm using collision algorithms from Erin Catto's paper I believe, so I do have some caching like saving axis/features for hull-hull collisions in order to resolve the next step faster.