Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917674256171008 |
|---|---|
| author | Guo, Yuxuan Peng, Shaohui Guo, Jiaming Huang, Di Zhang, Xishan Zhang, Rui Hao, Yifan Li, Ling Tian, Zikang Gao, Mingju Li, Yutai Gan, Yiming Liang, Shuai Zhang, Zihao Du, Zidong Guo, Qi Hu, Xing Chen, Yunji |
| author_facet | Guo, Yuxuan Peng, Shaohui Guo, Jiaming Huang, Di Zhang, Xishan Zhang, Rui Hao, Yifan Li, Ling Tian, Zikang Gao, Mingju Li, Yutai Gan, Yiming Liang, Shuai Zhang, Zihao Du, Zidong Guo, Qi Hu, Xing Chen, Yunji |
| contents | Building open agents has always been the ultimate goal in AI research, and creative agents are the more enticing. Existing LLM agents excel at long-horizon tasks with well-defined goals (e.g., `mine diamonds' in Minecraft). However, they encounter difficulties on creative tasks with open goals and abstract criteria due to the inability to bridge the gap between them, thus lacking feedback for self-improvement in solving the task. In this work, we introduce autonomous embodied verification techniques for agents to fill the gap, laying the groundwork for creative tasks. Specifically, we propose the Luban agent target creative building tasks in Minecraft, which equips with two-level autonomous embodied verification inspired by human design practices: (1) visual verification of 3D structural speculates, which comes from agent synthesized CAD modeling programs; (2) pragmatic verification of the creation by generating and verifying environment-relevant functionality programs based on the abstract criteria. Extensive multi-dimensional human studies and Elo ratings show that the Luban completes diverse creative building tasks in our proposed benchmark and outperforms other baselines ($33\%$ to $100\%$) in both visualization and pragmatism. Additional demos on the real-world robotic arm show the creation potential of the Luban in the physical world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15414 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification Guo, Yuxuan Peng, Shaohui Guo, Jiaming Huang, Di Zhang, Xishan Zhang, Rui Hao, Yifan Li, Ling Tian, Zikang Gao, Mingju Li, Yutai Gan, Yiming Liang, Shuai Zhang, Zihao Du, Zidong Guo, Qi Hu, Xing Chen, Yunji Artificial Intelligence Building open agents has always been the ultimate goal in AI research, and creative agents are the more enticing. Existing LLM agents excel at long-horizon tasks with well-defined goals (e.g., `mine diamonds' in Minecraft). However, they encounter difficulties on creative tasks with open goals and abstract criteria due to the inability to bridge the gap between them, thus lacking feedback for self-improvement in solving the task. In this work, we introduce autonomous embodied verification techniques for agents to fill the gap, laying the groundwork for creative tasks. Specifically, we propose the Luban agent target creative building tasks in Minecraft, which equips with two-level autonomous embodied verification inspired by human design practices: (1) visual verification of 3D structural speculates, which comes from agent synthesized CAD modeling programs; (2) pragmatic verification of the creation by generating and verifying environment-relevant functionality programs based on the abstract criteria. Extensive multi-dimensional human studies and Elo ratings show that the Luban completes diverse creative building tasks in our proposed benchmark and outperforms other baselines ($33\%$ to $100\%$) in both visualization and pragmatism. Additional demos on the real-world robotic arm show the creation potential of the Luban in the physical world. |
| title | Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2405.15414 |