Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917636435083264 |
|---|---|
| author | Zhao, Zhonghan Ma, Ke Chai, Wenhao Wang, Xuan Chen, Kewei Guo, Dongxu Zhang, Yanting Wang, Hongwei Wang, Gaoang |
| author_facet | Zhao, Zhonghan Ma, Ke Chai, Wenhao Wang, Xuan Chen, Kewei Guo, Dongxu Zhang, Yanting Wang, Hongwei Wang, Gaoang |
| contents | With the power of large language models (LLMs), open-ended embodied agents can flexibly understand human instructions, generate interpretable guidance strategies, and output executable actions. Nowadays, Multi-modal Language Models~(MLMs) integrate multi-modal signals into LLMs, further bringing richer perception to entity agents and allowing embodied agents to perceive world-understanding tasks more delicately. However, existing works: 1) operate independently by agents, each containing multiple LLMs, from perception to action, resulting in gaps between complex tasks and execution; 2) train MLMs on static data, struggling with dynamics in open-ended scenarios; 3) input prior knowledge directly as prompts, suppressing application flexibility. We propose STEVE-2, a hierarchical knowledge distillation framework for open-ended embodied tasks, characterized by 1) a hierarchical system for multi-granular task division, 2) a mirrored distillation method for parallel simulation data, and 3) an extra expert model for bringing additional knowledge into parallel simulation. After distillation, embodied agents can complete complex, open-ended tasks without additional expert guidance, utilizing the performance and knowledge of a versatile MLM. Extensive evaluations on navigation and creation tasks highlight the superior performance of STEVE-2 in open-ended tasks, with $1.4 \times$ - $7.3 \times$ in performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04619 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model Zhao, Zhonghan Ma, Ke Chai, Wenhao Wang, Xuan Chen, Kewei Guo, Dongxu Zhang, Yanting Wang, Hongwei Wang, Gaoang Artificial Intelligence Computer Vision and Pattern Recognition With the power of large language models (LLMs), open-ended embodied agents can flexibly understand human instructions, generate interpretable guidance strategies, and output executable actions. Nowadays, Multi-modal Language Models~(MLMs) integrate multi-modal signals into LLMs, further bringing richer perception to entity agents and allowing embodied agents to perceive world-understanding tasks more delicately. However, existing works: 1) operate independently by agents, each containing multiple LLMs, from perception to action, resulting in gaps between complex tasks and execution; 2) train MLMs on static data, struggling with dynamics in open-ended scenarios; 3) input prior knowledge directly as prompts, suppressing application flexibility. We propose STEVE-2, a hierarchical knowledge distillation framework for open-ended embodied tasks, characterized by 1) a hierarchical system for multi-granular task division, 2) a mirrored distillation method for parallel simulation data, and 3) an extra expert model for bringing additional knowledge into parallel simulation. After distillation, embodied agents can complete complex, open-ended tasks without additional expert guidance, utilizing the performance and knowledge of a versatile MLM. Extensive evaluations on navigation and creation tasks highlight the superior performance of STEVE-2 in open-ended tasks, with $1.4 \times$ - $7.3 \times$ in performance. |
| title | Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.04619 |