I think I'm making some progress with queueing up the updates in case they are early, and applying them at their designated time. It definitely removed the worst of the jitter. However, there's some inherent problems with this:
* Queuing up = introducing latency. Not much to do about this.
* How large queue should one allow? I've set an arbitrary number right now (3!) but I think it needs some more thought...
* What happens to packets that are too late? I think I'll try extrapolating the position in case this happens.