Embodied Long Horizon Manipulation with Closed-loop Code Generation and Incremental Few-shot Adaptation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918128105029632 |
|---|---|
| author | Meng, Yuan Yao, Xiangtong Ye, Haihui Zhou, Yirui Zhang, Shengqiang Sun, Zhenguo Li, Xukun Bing, Zhenshan Knoll, Alois |
| author_facet | Meng, Yuan Yao, Xiangtong Ye, Haihui Zhou, Yirui Zhang, Shengqiang Sun, Zhenguo Li, Xukun Bing, Zhenshan Knoll, Alois |
| contents | Embodied long-horizon manipulation requires robotic systems to process multimodal inputs-such as vision and natural language-and translate them into executable actions. However, existing learning-based approaches often depend on large, task-specific datasets and struggle to generalize to unseen scenarios. Recent methods have explored using large language models (LLMs) as high-level planners that decompose tasks into subtasks using natural language and guide pretrained low-level controllers. Yet, these approaches assume perfect execution from low-level policies, which is unrealistic in real-world environments with noise or suboptimal behaviors. To overcome this, we fully discard the pretrained low-level policy and instead use the LLM to directly generate executable code plans within a closed-loop framework. Our planner employs chain-of-thought (CoT)-guided few-shot learning with incrementally structured examples to produce robust and generalizable task plans. Complementing this, a reporter evaluates outcomes using RGB-D and delivers structured feedback, enabling recovery from misalignment and replanning under partial observability. This design eliminates per-step inference, reduces computational overhead, and limits error accumulation that was observed in previous methods. Our framework achieves state-of-the-art performance on 30+ diverse seen and unseen long-horizon tasks across LoHoRavens, CALVIN, Franka Kitchen, and cluttered real-world settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21969 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Embodied Long Horizon Manipulation with Closed-loop Code Generation and Incremental Few-shot Adaptation Meng, Yuan Yao, Xiangtong Ye, Haihui Zhou, Yirui Zhang, Shengqiang Sun, Zhenguo Li, Xukun Bing, Zhenshan Knoll, Alois Robotics Artificial Intelligence Embodied long-horizon manipulation requires robotic systems to process multimodal inputs-such as vision and natural language-and translate them into executable actions. However, existing learning-based approaches often depend on large, task-specific datasets and struggle to generalize to unseen scenarios. Recent methods have explored using large language models (LLMs) as high-level planners that decompose tasks into subtasks using natural language and guide pretrained low-level controllers. Yet, these approaches assume perfect execution from low-level policies, which is unrealistic in real-world environments with noise or suboptimal behaviors. To overcome this, we fully discard the pretrained low-level policy and instead use the LLM to directly generate executable code plans within a closed-loop framework. Our planner employs chain-of-thought (CoT)-guided few-shot learning with incrementally structured examples to produce robust and generalizable task plans. Complementing this, a reporter evaluates outcomes using RGB-D and delivers structured feedback, enabling recovery from misalignment and replanning under partial observability. This design eliminates per-step inference, reduces computational overhead, and limits error accumulation that was observed in previous methods. Our framework achieves state-of-the-art performance on 30+ diverse seen and unseen long-horizon tasks across LoHoRavens, CALVIN, Franka Kitchen, and cluttered real-world settings. |
| title | Embodied Long Horizon Manipulation with Closed-loop Code Generation and Incremental Few-shot Adaptation |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2503.21969 |