Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

arXiv cs.GR details PRISM, a real-to-sim-to-real framework aimed at scaling humanoid loco-manipulation from a tiny set of real interaction videos. The core idea is to amplify a handful of clean clips into hundreds of diverse counterfactual human-object interactions, then convert those videos into physically plausible motion for training.

That matters because high-quality embodied data is expensive: you need full-body visibility, clear object contact, and enough variation to cover the long tail of object shapes and starting states. PRISM tackles that bottleneck with video-to-video generation and a contact-anchored reconstruction step that retargets the synthetic motion into sim-ready trajectories.

The practical result is a single policy that generalizes across unseen instances within an object category. In the demonstrated setup, a humanoid robot used only onboard depth observations to pick up, carry, and drop boxes, barrels, bins, and balls, with no real-world fine-tuning.

For robotics teams, the interesting part is the data strategy: instead of hand-collecting massive teleop datasets, the pipeline tries to manufacture diversity from a small seed set. If it holds up beyond the showcased categories, it could lower the cost of training generalist manipulation policies and make sim-to-real transfer less dependent on exhaustive real-world capture.