Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912263043022848 |
|---|---|
| author | Zhi, Peiyuan Zhang, Zhiyuan Zhao, Yu Han, Muzhi Zhang, Zeyu Li, Zhitian Jiao, Ziyuan Jia, Baoxiong Huang, Siyuan |
| author_facet | Zhi, Peiyuan Zhang, Zhiyuan Zhao, Yu Han, Muzhi Zhang, Zeyu Li, Zhitian Jiao, Ziyuan Jia, Baoxiong Huang, Siyuan |
| contents | Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback. In this work, we present COME-robot, the first closed-loop robotic system utilizing the GPT-4V vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios.COME-robot incorporates two key innovative modules: (i) a multi-level open-vocabulary perception and situated reasoning module that enables effective exploration of the 3D environment and target object identification using commonsense knowledge and situated information, and (ii) an iterative closed-loop feedback and restoration mechanism that verifies task feasibility, monitors execution success, and traces failure causes across different modules for robust failure recovery. Through comprehensive experiments involving 8 challenging real-world mobile and tabletop manipulation tasks, COME-robot demonstrates a significant improvement in task success rate (~35%) compared to state-of-the-art methods. We further conduct comprehensive analyses to elucidate how COME-robot's design facilitates failure recovery, free-form instruction following, and long-horizon task planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_10220 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V Zhi, Peiyuan Zhang, Zhiyuan Zhao, Yu Han, Muzhi Zhang, Zeyu Li, Zhitian Jiao, Ziyuan Jia, Baoxiong Huang, Siyuan Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback. In this work, we present COME-robot, the first closed-loop robotic system utilizing the GPT-4V vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios.COME-robot incorporates two key innovative modules: (i) a multi-level open-vocabulary perception and situated reasoning module that enables effective exploration of the 3D environment and target object identification using commonsense knowledge and situated information, and (ii) an iterative closed-loop feedback and restoration mechanism that verifies task feasibility, monitors execution success, and traces failure causes across different modules for robust failure recovery. Through comprehensive experiments involving 8 challenging real-world mobile and tabletop manipulation tasks, COME-robot demonstrates a significant improvement in task success rate (~35%) compared to state-of-the-art methods. We further conduct comprehensive analyses to elucidate how COME-robot's design facilitates failure recovery, free-form instruction following, and long-horizon task planning. |
| title | Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2404.10220 |