Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929727308038144 |
|---|---|
| author | Feng, Yunhai Han, Jiaming Yang, Zhuoran Yue, Xiangyu Levine, Sergey Luo, Jianlan |
| author_facet | Feng, Yunhai Han, Jiaming Yang, Zhuoran Yue, Xiangyu Levine, Sergey Luo, Jianlan |
| contents | Solving complex long-horizon robotic manipulation problems requires sophisticated high-level planning capabilities, the ability to reason about the physical world, and reactively choose appropriate motor skills. Vision-language models (VLMs) pretrained on Internet data could in principle offer a framework for tackling such problems. However, in their current form, VLMs lack both the nuanced understanding of intricate physics required for robotic manipulation and the ability to reason over long horizons to address error compounding issues. In this paper, we introduce a novel test-time computation framework that enhances VLMs' physical reasoning capabilities for multi-stage manipulation tasks. At its core, our approach iteratively improves a pretrained VLM with a "reflection" mechanism - it uses a generative model to imagine future world states, leverages these predictions to guide action selection, and critically reflects on potential suboptimalities to refine its reasoning. Experimental results demonstrate that our method significantly outperforms several state-of-the-art commercial VLMs as well as other post-training approaches such as Monte Carlo Tree Search (MCTS). Videos are available at https://reflect-vlm.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16707 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation Feng, Yunhai Han, Jiaming Yang, Zhuoran Yue, Xiangyu Levine, Sergey Luo, Jianlan Robotics Artificial Intelligence Machine Learning Solving complex long-horizon robotic manipulation problems requires sophisticated high-level planning capabilities, the ability to reason about the physical world, and reactively choose appropriate motor skills. Vision-language models (VLMs) pretrained on Internet data could in principle offer a framework for tackling such problems. However, in their current form, VLMs lack both the nuanced understanding of intricate physics required for robotic manipulation and the ability to reason over long horizons to address error compounding issues. In this paper, we introduce a novel test-time computation framework that enhances VLMs' physical reasoning capabilities for multi-stage manipulation tasks. At its core, our approach iteratively improves a pretrained VLM with a "reflection" mechanism - it uses a generative model to imagine future world states, leverages these predictions to guide action selection, and critically reflects on potential suboptimalities to refine its reasoning. Experimental results demonstrate that our method significantly outperforms several state-of-the-art commercial VLMs as well as other post-training approaches such as Monte Carlo Tree Search (MCTS). Videos are available at https://reflect-vlm.github.io. |
| title | Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation |
| topic | Robotics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.16707 |