Sim-and-Human Co-training for Data-Efficient and Generalizable Robotic Manipulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911402419027968 |
|---|---|
| author | Fang, Kaipeng Liang, Weiqing Li, Yuyang Zhang, Ji Zeng, Pengpeng Gao, Lianli Song, Jingkuan Shen, Heng Tao |
| author_facet | Fang, Kaipeng Liang, Weiqing Li, Yuyang Zhang, Ji Zeng, Pengpeng Gao, Lianli Song, Jingkuan Shen, Heng Tao |
| contents | Synthetic simulation data and real-world human data provide scalable alternatives to circumvent the prohibitive costs of robot data collection. However, these sources suffer from the sim-to-real visual gap and the human-to-robot embodiment gap, respectively, which limits the policy's generalization to real-world scenarios. In this work, we identify a natural yet underexplored complementarity between these sources: simulation offers the robot action that human data lacks, while human data provides the real-world observation that simulation struggles to render. Motivated by this insight, we present SimHum, a co-training framework to simultaneously extract kinematic prior from simulated robot actions and visual prior from real-world human observations. Based on the two complementary priors, we achieve data-efficient and generalizable robotic manipulation in real-world tasks. Empirically, SimHum outperforms the baseline by up to $\mathbf{40\%}$ under the same data collection budget, and achieves a $\mathbf{62.5\%}$ OOD success with only 80 real data, outperforming the real only baseline by $7.1\times$. Videos and additional information can be found at \href{https://kaipengfang.github.io/sim-and-human}{project website}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_19406 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Sim-and-Human Co-training for Data-Efficient and Generalizable Robotic Manipulation Fang, Kaipeng Liang, Weiqing Li, Yuyang Zhang, Ji Zeng, Pengpeng Gao, Lianli Song, Jingkuan Shen, Heng Tao Robotics Artificial Intelligence Synthetic simulation data and real-world human data provide scalable alternatives to circumvent the prohibitive costs of robot data collection. However, these sources suffer from the sim-to-real visual gap and the human-to-robot embodiment gap, respectively, which limits the policy's generalization to real-world scenarios. In this work, we identify a natural yet underexplored complementarity between these sources: simulation offers the robot action that human data lacks, while human data provides the real-world observation that simulation struggles to render. Motivated by this insight, we present SimHum, a co-training framework to simultaneously extract kinematic prior from simulated robot actions and visual prior from real-world human observations. Based on the two complementary priors, we achieve data-efficient and generalizable robotic manipulation in real-world tasks. Empirically, SimHum outperforms the baseline by up to $\mathbf{40\%}$ under the same data collection budget, and achieves a $\mathbf{62.5\%}$ OOD success with only 80 real data, outperforming the real only baseline by $7.1\times$. Videos and additional information can be found at \href{https://kaipengfang.github.io/sim-and-human}{project website}. |
| title | Sim-and-Human Co-training for Data-Efficient and Generalizable Robotic Manipulation |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2601.19406 |