Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914689071448064 |
|---|---|
| author | Liu, Jinyi Yuan, Yifu Hao, Jianye Ni, Fei Fu, Lingzhi Chen, Yibin Zheng, Yan |
| author_facet | Liu, Jinyi Yuan, Yifu Hao, Jianye Ni, Fei Fu, Lingzhi Chen, Yibin Zheng, Yan |
| contents | Recently, there has been considerable attention towards leveraging large language models (LLMs) to enhance decision-making processes. However, aligning the natural language text instructions generated by LLMs with the vectorized operations required for execution presents a significant challenge, often necessitating task-specific details. To circumvent the need for such task-specific granularity, inspired by preference-based policy learning approaches, we investigate the utilization of multimodal LLMs to provide automated preference feedback solely from image inputs to guide decision-making. In this study, we train a multimodal LLM, termed CriticGPT, capable of understanding trajectory videos in robot manipulation tasks, serving as a critic to offer analysis and preference feedback. Subsequently, we validate the effectiveness of preference labels generated by CriticGPT from a reward modeling perspective. Experimental evaluation of the algorithm's preference accuracy demonstrates its effective generalization ability to new tasks. Furthermore, performance on Meta-World tasks reveals that CriticGPT's reward model efficiently guides policy learning, surpassing rewards based on state-of-the-art pre-trained representation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14245 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models Liu, Jinyi Yuan, Yifu Hao, Jianye Ni, Fei Fu, Lingzhi Chen, Yibin Zheng, Yan Robotics Artificial Intelligence Machine Learning Recently, there has been considerable attention towards leveraging large language models (LLMs) to enhance decision-making processes. However, aligning the natural language text instructions generated by LLMs with the vectorized operations required for execution presents a significant challenge, often necessitating task-specific details. To circumvent the need for such task-specific granularity, inspired by preference-based policy learning approaches, we investigate the utilization of multimodal LLMs to provide automated preference feedback solely from image inputs to guide decision-making. In this study, we train a multimodal LLM, termed CriticGPT, capable of understanding trajectory videos in robot manipulation tasks, serving as a critic to offer analysis and preference feedback. Subsequently, we validate the effectiveness of preference labels generated by CriticGPT from a reward modeling perspective. Experimental evaluation of the algorithm's preference accuracy demonstrates its effective generalization ability to new tasks. Furthermore, performance on Meta-World tasks reveals that CriticGPT's reward model efficiently guides policy learning, surpassing rewards based on state-of-the-art pre-trained representation models. |
| title | Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models |
| topic | Robotics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2402.14245 |