LEGENT: Open Platform for Embodied Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917751734403072 |
|---|---|
| author | Cheng, Zhili Wang, Zhitong Hu, Jinyi Hu, Shengding Liu, An Tu, Yuge Li, Pengkai Shi, Lei Liu, Zhiyuan Sun, Maosong |
| author_facet | Cheng, Zhili Wang, Zhitong Hu, Jinyi Hu, Shengding Liu, An Tu, Yuge Li, Pengkai Shi, Lei Liu, Zhiyuan Sun, Maosong |
| contents | Despite advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), their integration into language-grounded, human-like embodied agents remains incomplete, hindering complex real-life task performance in physical environments. Existing integrations often feature limited open sourcing, challenging collective progress in this field. We introduce LEGENT, an open, scalable platform for developing embodied agents using LLMs and LMMs. LEGENT offers a dual approach: a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface, and a sophisticated data generation pipeline utilizing advanced algorithms to exploit supervision from simulated worlds at scale. In our experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks, showcasing promising generalization capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_18243 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LEGENT: Open Platform for Embodied Agents Cheng, Zhili Wang, Zhitong Hu, Jinyi Hu, Shengding Liu, An Tu, Yuge Li, Pengkai Shi, Lei Liu, Zhiyuan Sun, Maosong Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Robotics Despite advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), their integration into language-grounded, human-like embodied agents remains incomplete, hindering complex real-life task performance in physical environments. Existing integrations often feature limited open sourcing, challenging collective progress in this field. We introduce LEGENT, an open, scalable platform for developing embodied agents using LLMs and LMMs. LEGENT offers a dual approach: a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface, and a sophisticated data generation pipeline utilizing advanced algorithms to exploit supervision from simulated worlds at scale. In our experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks, showcasing promising generalization capabilities. |
| title | LEGENT: Open Platform for Embodied Agents |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Robotics |
| url | https://arxiv.org/abs/2404.18243 |