HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909723906801664 |
|---|---|
| author | Liu, Yibin Liang, Zhixuan Chen, Zanxin Chen, Tianxing Hu, Mengkang Dong, Wanxi Xu, Congsheng Han, Zhaoming Qin, Yusen Mu, Yao |
| author_facet | Liu, Yibin Liang, Zhixuan Chen, Zanxin Chen, Tianxing Hu, Mengkang Dong, Wanxi Xu, Congsheng Han, Zhaoming Qin, Yusen Mu, Yao |
| contents | Recent advances in multimodal large language models (MLLMs) have enabled richer perceptual grounding for code policy generation in embodied agents. However, most existing systems lack effective mechanisms to adaptively monitor policy execution and repair codes during task completion. In this work, we introduce HyCodePolicy, a hybrid language-based control framework that systematically integrates code synthesis, geometric grounding, perceptual monitoring, and iterative repair into a closed-loop programming cycle for embodied agents. Technically, given a natural language instruction, our system first decomposes it into subgoals and generates an initial executable program grounded in object-centric geometric primitives. The program is then executed in simulation, while a vision-language model (VLM) observes selected checkpoints to detect and localize execution failures and infer failure reasons. By fusing structured execution traces capturing program-level events with VLM-based perceptual feedback, HyCodePolicy infers failure causes and repairs programs. This hybrid dual feedback mechanism enables self-correcting program synthesis with minimal human supervision. Our results demonstrate that HyCodePolicy significantly improves the robustness and sample efficiency of robot manipulation policies, offering a scalable strategy for integrating multimodal reasoning into autonomous decision-making pipelines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_02629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents Liu, Yibin Liang, Zhixuan Chen, Zanxin Chen, Tianxing Hu, Mengkang Dong, Wanxi Xu, Congsheng Han, Zhaoming Qin, Yusen Mu, Yao Robotics Artificial Intelligence Computation and Language Recent advances in multimodal large language models (MLLMs) have enabled richer perceptual grounding for code policy generation in embodied agents. However, most existing systems lack effective mechanisms to adaptively monitor policy execution and repair codes during task completion. In this work, we introduce HyCodePolicy, a hybrid language-based control framework that systematically integrates code synthesis, geometric grounding, perceptual monitoring, and iterative repair into a closed-loop programming cycle for embodied agents. Technically, given a natural language instruction, our system first decomposes it into subgoals and generates an initial executable program grounded in object-centric geometric primitives. The program is then executed in simulation, while a vision-language model (VLM) observes selected checkpoints to detect and localize execution failures and infer failure reasons. By fusing structured execution traces capturing program-level events with VLM-based perceptual feedback, HyCodePolicy infers failure causes and repairs programs. This hybrid dual feedback mechanism enables self-correcting program synthesis with minimal human supervision. Our results demonstrate that HyCodePolicy significantly improves the robustness and sample efficiency of robot manipulation policies, offering a scalable strategy for integrating multimodal reasoning into autonomous decision-making pipelines. |
| title | HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents |
| topic | Robotics Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2508.02629 |