STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909225288990720 |
|---|---|
| author | Zhao, Zhonghan Chai, Wenhao Wang, Xuan Ma, Ke Chen, Kewei Guo, Dongxu Ye, Tian Zhang, Yanting Wang, Hongwei Wang, Gaoang |
| author_facet | Zhao, Zhonghan Chai, Wenhao Wang, Xuan Ma, Ke Chen, Kewei Guo, Dongxu Ye, Tian Zhang, Yanting Wang, Hongwei Wang, Gaoang |
| contents | Building an embodied agent system with a large language model (LLM) as its core is a promising direction. Due to the significant costs and uncontrollable factors associated with deploying and training such agents in the real world, we have decided to begin our exploration within the Minecraft environment. Our STEVE Series agents can complete basic tasks in a virtual environment and more challenging tasks such as navigation and even creative tasks, with an efficiency far exceeding previous state-of-the-art methods by a factor of $2.5\times$ to $7.3\times$. We begin our exploration with a vanilla large language model, augmenting it with a vision encoder and an action codebase trained on our collected high-quality dataset STEVE-21K. Subsequently, we enhanced it with a Critic and memory to transform it into a complex system. Finally, we constructed a hierarchical multi-agent system. Our recent work explored how to prune the agent system through knowledge distillation. In the future, we will explore more potential applications of STEVE agents in the real world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11247 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft Zhao, Zhonghan Chai, Wenhao Wang, Xuan Ma, Ke Chen, Kewei Guo, Dongxu Ye, Tian Zhang, Yanting Wang, Hongwei Wang, Gaoang Computer Vision and Pattern Recognition Building an embodied agent system with a large language model (LLM) as its core is a promising direction. Due to the significant costs and uncontrollable factors associated with deploying and training such agents in the real world, we have decided to begin our exploration within the Minecraft environment. Our STEVE Series agents can complete basic tasks in a virtual environment and more challenging tasks such as navigation and even creative tasks, with an efficiency far exceeding previous state-of-the-art methods by a factor of $2.5\times$ to $7.3\times$. We begin our exploration with a vanilla large language model, augmenting it with a vision encoder and an action codebase trained on our collected high-quality dataset STEVE-21K. Subsequently, we enhanced it with a Critic and memory to transform it into a complex system. Finally, we constructed a hierarchical multi-agent system. Our recent work explored how to prune the agent system through knowledge distillation. In the future, we will explore more potential applications of STEVE agents in the real world. |
| title | STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.11247 |