Object-Centric Instruction Augmentation for Robotic Manipulation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911768687673344 |
|---|---|
| author | Wen, Junjie Zhu, Yichen Zhu, Minjie Li, Jinming Xu, Zhiyuan Che, Zhengping Shen, Chaomin Peng, Yaxin Liu, Dong Feng, Feifei Tang, Jian |
| author_facet | Wen, Junjie Zhu, Yichen Zhu, Minjie Li, Jinming Xu, Zhiyuan Che, Zhengping Shen, Chaomin Peng, Yaxin Liu, Dong Feng, Feifei Tang, Jian |
| contents | Humans interpret scenes by recognizing both the identities and positions of objects in their observations. For a robot to perform tasks such as \enquote{pick and place}, understanding both what the objects are and where they are located is crucial. While the former has been extensively discussed in the literature that uses the large language model to enrich the text descriptions, the latter remains underexplored. In this work, we introduce the \textit{Object-Centric Instruction Augmentation (OCI)} framework to augment highly semantic and information-dense language instruction with position cues. We utilize a Multi-modal Large Language Model (MLLM) to weave knowledge of object locations into natural language instruction, thus aiding the policy network in mastering actions for versatile manipulation. Additionally, we present a feature reuse mechanism to integrate the vision-language features from off-the-shelf pre-trained MLLM into policy networks. Through a series of simulated and real-world robotic tasks, we demonstrate that robotic manipulator imitation policies trained with our enhanced instructions outperform those relying solely on traditional language instructions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02814 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Object-Centric Instruction Augmentation for Robotic Manipulation Wen, Junjie Zhu, Yichen Zhu, Minjie Li, Jinming Xu, Zhiyuan Che, Zhengping Shen, Chaomin Peng, Yaxin Liu, Dong Feng, Feifei Tang, Jian Robotics Computer Vision and Pattern Recognition Humans interpret scenes by recognizing both the identities and positions of objects in their observations. For a robot to perform tasks such as \enquote{pick and place}, understanding both what the objects are and where they are located is crucial. While the former has been extensively discussed in the literature that uses the large language model to enrich the text descriptions, the latter remains underexplored. In this work, we introduce the \textit{Object-Centric Instruction Augmentation (OCI)} framework to augment highly semantic and information-dense language instruction with position cues. We utilize a Multi-modal Large Language Model (MLLM) to weave knowledge of object locations into natural language instruction, thus aiding the policy network in mastering actions for versatile manipulation. Additionally, we present a feature reuse mechanism to integrate the vision-language features from off-the-shelf pre-trained MLLM into policy networks. Through a series of simulated and real-world robotic tasks, we demonstrate that robotic manipulator imitation policies trained with our enhanced instructions outperform those relying solely on traditional language instructions. |
| title | Object-Centric Instruction Augmentation for Robotic Manipulation |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.02814 |