Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914962698403840 |
|---|---|
| author | Li, Jianxiong Wang, Zhihao Zheng, Jinliang Zhou, Xiaoai Wang, Guanming Song, Guanglu Liu, Yu Liu, Jingjing Zhang, Ya-Qin Yu, Junzhi Zhan, Xianyuan |
| author_facet | Li, Jianxiong Wang, Zhihao Zheng, Jinliang Zhou, Xiaoai Wang, Guanming Song, Guanglu Liu, Yu Liu, Jingjing Zhang, Ya-Qin Yu, Junzhi Zhan, Xianyuan |
| contents | Multimodal task specification is essential for enhanced robotic performance, where \textit{Cross-modality Alignment} enables the robot to holistically understand complex task instructions. Directly annotating multimodal instructions for model training proves impractical, due to the sparsity of paired multimodal data. In this study, we demonstrate that by leveraging unimodal instructions abundant in real data, we can effectively teach robots to learn multimodal task specifications. First, we endow the robot with strong \textit{Cross-modality Alignment} capabilities, by pretraining a robotic multimodal encoder using extensive out-of-domain data. Then, we employ two Collapse and Corrupt operations to further bridge the remaining modality gap in the learned multimodal representation. This approach projects different modalities of identical task goal as interchangeable representations, thus enabling accurate robotic operations within a well-aligned multimodal latent space. Evaluation across more than 130 tasks and 4000 evaluations on both simulated LIBERO benchmark and real robot platforms showcases the superior capabilities of our proposed framework, demonstrating significant advantage in overcoming data constraints in robotic learning. Website: zh1hao.wang/Robo_MUTUAL |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01529 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning Li, Jianxiong Wang, Zhihao Zheng, Jinliang Zhou, Xiaoai Wang, Guanming Song, Guanglu Liu, Yu Liu, Jingjing Zhang, Ya-Qin Yu, Junzhi Zhan, Xianyuan Robotics Computer Vision and Pattern Recognition Multimodal task specification is essential for enhanced robotic performance, where \textit{Cross-modality Alignment} enables the robot to holistically understand complex task instructions. Directly annotating multimodal instructions for model training proves impractical, due to the sparsity of paired multimodal data. In this study, we demonstrate that by leveraging unimodal instructions abundant in real data, we can effectively teach robots to learn multimodal task specifications. First, we endow the robot with strong \textit{Cross-modality Alignment} capabilities, by pretraining a robotic multimodal encoder using extensive out-of-domain data. Then, we employ two Collapse and Corrupt operations to further bridge the remaining modality gap in the learned multimodal representation. This approach projects different modalities of identical task goal as interchangeable representations, thus enabling accurate robotic operations within a well-aligned multimodal latent space. Evaluation across more than 130 tasks and 4000 evaluations on both simulated LIBERO benchmark and real robot platforms showcases the superior capabilities of our proposed framework, demonstrating significant advantage in overcoming data constraints in robotic learning. Website: zh1hao.wang/Robo_MUTUAL |
| title | Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.01529 |