MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912614708150272 |
|---|---|
| author | Li, Haoyun Zhang, Ivan Ouyang, Runqi Wang, Xiaofeng Zhu, Zheng Yang, Zhiqin Zhang, Zhentao Wang, Boyuan Ni, Chaojun Qin, Wenkang Chen, Xinze Ye, Yun Huang, Guan Song, Zhenbo Wang, Xingang |
| author_facet | Li, Haoyun Zhang, Ivan Ouyang, Runqi Wang, Xiaofeng Zhu, Zheng Yang, Zhiqin Zhang, Zhentao Wang, Boyuan Ni, Chaojun Qin, Wenkang Chen, Xinze Ye, Yun Huang, Guan Song, Zhenbo Wang, Xingang |
| contents | Vision Language Action (VLA) models derive their generalization capability from diverse training data, yet collecting embodied robot interaction data remains prohibitively expensive. In contrast, human demonstration videos are far more scalable and cost-efficient to collect, and recent studies confirm their effectiveness in training VLA models. However, a significant domain gap persists between human videos and robot-executed videos, including unstable camera viewpoints, visual discrepancies between human hands and robotic arms, and differences in motion dynamics. To bridge this gap, we propose MimicDreamer, a framework that turns fast, low-cost human demonstrations into robot-usable supervision by jointly aligning vision, viewpoint, and actions to directly support policy training. For visual alignment, we propose H2R Aligner, a video diffusion model that generates high-fidelity robot demonstration videos by transferring motion from human manipulation footage. For viewpoint stabilization, EgoStabilizer is proposed, which canonicalizes egocentric videos via homography and inpaints occlusions and distortions caused by warping. For action alignment, we map human hand trajectories to the robot frame and apply a constrained inverse kinematics solver to produce feasible, low-jitter joint commands with accurate pose tracking. Empirically, VLA models trained purely on our synthesized human-to-robot videos achieve few-shot execution on real robots. Moreover, scaling training with human data significantly boosts performance compared to models trained solely on real robot data; our approach improves the average success rate by 14.7\% across six representative manipulation tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22199 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training Li, Haoyun Zhang, Ivan Ouyang, Runqi Wang, Xiaofeng Zhu, Zheng Yang, Zhiqin Zhang, Zhentao Wang, Boyuan Ni, Chaojun Qin, Wenkang Chen, Xinze Ye, Yun Huang, Guan Song, Zhenbo Wang, Xingang Robotics Artificial Intelligence Vision Language Action (VLA) models derive their generalization capability from diverse training data, yet collecting embodied robot interaction data remains prohibitively expensive. In contrast, human demonstration videos are far more scalable and cost-efficient to collect, and recent studies confirm their effectiveness in training VLA models. However, a significant domain gap persists between human videos and robot-executed videos, including unstable camera viewpoints, visual discrepancies between human hands and robotic arms, and differences in motion dynamics. To bridge this gap, we propose MimicDreamer, a framework that turns fast, low-cost human demonstrations into robot-usable supervision by jointly aligning vision, viewpoint, and actions to directly support policy training. For visual alignment, we propose H2R Aligner, a video diffusion model that generates high-fidelity robot demonstration videos by transferring motion from human manipulation footage. For viewpoint stabilization, EgoStabilizer is proposed, which canonicalizes egocentric videos via homography and inpaints occlusions and distortions caused by warping. For action alignment, we map human hand trajectories to the robot frame and apply a constrained inverse kinematics solver to produce feasible, low-jitter joint commands with accurate pose tracking. Empirically, VLA models trained purely on our synthesized human-to-robot videos achieve few-shot execution on real robots. Moreover, scaling training with human data significantly boosts performance compared to models trained solely on real robot data; our approach improves the average success rate by 14.7\% across six representative manipulation tasks. |
| title | MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2509.22199 |