RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911461650989056 |
|---|---|
| author | Wu, Kun Hou, Chengkai Liu, Jiaming Che, Zhengping Ju, Xiaozhu Yang, Zhuqin Li, Meng Zhao, Yinuo Xu, Zhiyuan Yang, Guang Fan, Shichao Wang, Xinhua Liao, Fei Zhao, Zhen Li, Guangyu Jin, Zhao Wang, Lecheng Mao, Jilei Liu, Ning Ren, Pei Zhang, Qiang Lyu, Yaoxu Liu, Mengzhen He, Jingyang Luo, Yulin Gao, Zeyu Li, Chenxuan Gu, Chenyang Fu, Yankai Wu, Di Wang, Xingyu Chen, Sixiang Wang, Zhenyu An, Pengju Qian, Siyuan Zhang, Shanghang Tang, Jian |
| author_facet | Wu, Kun Hou, Chengkai Liu, Jiaming Che, Zhengping Ju, Xiaozhu Yang, Zhuqin Li, Meng Zhao, Yinuo Xu, Zhiyuan Yang, Guang Fan, Shichao Wang, Xinhua Liao, Fei Zhao, Zhen Li, Guangyu Jin, Zhao Wang, Lecheng Mao, Jilei Liu, Ning Ren, Pei Zhang, Qiang Lyu, Yaoxu Liu, Mengzhen He, Jingyang Luo, Yulin Gao, Zeyu Li, Chenxuan Gu, Chenyang Fu, Yankai Wu, Di Wang, Xingyu Chen, Sixiang Wang, Zhenyu An, Pengju Qian, Siyuan Zhang, Shanghang Tang, Jian |
| contents | In this paper, we introduce RoboMIND (Multi-embodiment Intelligence Normative Data for Robot Manipulation), a dataset containing 107k demonstration trajectories across 479 diverse tasks involving 96 object classes. RoboMIND is collected through human teleoperation and encompasses comprehensive robotic-related information, including multi-view observations, proprioceptive robot state information, and linguistic task descriptions. To ensure data consistency and reliability for imitation learning, RoboMIND is built on a unified data collection platform and a standardized protocol, covering four distinct robotic embodiments: the Franka Emika Panda, the UR5e, the AgileX dual-arm robot, and a humanoid robot with dual dexterous hands. Our dataset also includes 5k real-world failure demonstrations, each accompanied by detailed causes, enabling failure reflection and correction during policy learning. Additionally, we created a digital twin environment in the Isaac Sim simulator, replicating the real-world tasks and assets, which facilitates the low-cost collection of additional training data and enables efficient evaluation. To demonstrate the quality and diversity of our dataset, we conducted extensive experiments using various imitation learning methods for single-task settings and state-of-the-art Vision-Language-Action (VLA) models for multi-task scenarios. By leveraging RoboMIND, the VLA models achieved high manipulation success rates and demonstrated strong generalization capabilities. To the best of our knowledge, RoboMIND is the largest multi-embodiment teleoperation dataset collected on a unified platform, providing large-scale and high-quality robotic training data. Our project is at https://x-humanoid-robomind.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13877 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation Wu, Kun Hou, Chengkai Liu, Jiaming Che, Zhengping Ju, Xiaozhu Yang, Zhuqin Li, Meng Zhao, Yinuo Xu, Zhiyuan Yang, Guang Fan, Shichao Wang, Xinhua Liao, Fei Zhao, Zhen Li, Guangyu Jin, Zhao Wang, Lecheng Mao, Jilei Liu, Ning Ren, Pei Zhang, Qiang Lyu, Yaoxu Liu, Mengzhen He, Jingyang Luo, Yulin Gao, Zeyu Li, Chenxuan Gu, Chenyang Fu, Yankai Wu, Di Wang, Xingyu Chen, Sixiang Wang, Zhenyu An, Pengju Qian, Siyuan Zhang, Shanghang Tang, Jian Robotics Artificial Intelligence In this paper, we introduce RoboMIND (Multi-embodiment Intelligence Normative Data for Robot Manipulation), a dataset containing 107k demonstration trajectories across 479 diverse tasks involving 96 object classes. RoboMIND is collected through human teleoperation and encompasses comprehensive robotic-related information, including multi-view observations, proprioceptive robot state information, and linguistic task descriptions. To ensure data consistency and reliability for imitation learning, RoboMIND is built on a unified data collection platform and a standardized protocol, covering four distinct robotic embodiments: the Franka Emika Panda, the UR5e, the AgileX dual-arm robot, and a humanoid robot with dual dexterous hands. Our dataset also includes 5k real-world failure demonstrations, each accompanied by detailed causes, enabling failure reflection and correction during policy learning. Additionally, we created a digital twin environment in the Isaac Sim simulator, replicating the real-world tasks and assets, which facilitates the low-cost collection of additional training data and enables efficient evaluation. To demonstrate the quality and diversity of our dataset, we conducted extensive experiments using various imitation learning methods for single-task settings and state-of-the-art Vision-Language-Action (VLA) models for multi-task scenarios. By leveraging RoboMIND, the VLA models achieved high manipulation success rates and demonstrated strong generalization capabilities. To the best of our knowledge, RoboMIND is the largest multi-embodiment teleoperation dataset collected on a unified platform, providing large-scale and high-quality robotic training data. Our project is at https://x-humanoid-robomind.github.io/. |
| title | RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2412.13877 |