Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912703867518976 |
|---|---|
| author | Tan, Weihao Li, Xiangyang Fang, Yunhao Yao, Heyuan Yan, Shi Luo, Hao Ao, Tenglong Li, Huihui Ren, Hongbin Yi, Bairen Qin, Yujia An, Bo Liu, Libin Shi, Guang |
| author_facet | Tan, Weihao Li, Xiangyang Fang, Yunhao Yao, Heyuan Yan, Shi Luo, Hao Ao, Tenglong Li, Huihui Ren, Hongbin Yi, Bairen Qin, Yujia An, Bo Liu, Libin Shi, Guang |
| contents | We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_08892 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds Tan, Weihao Li, Xiangyang Fang, Yunhao Yao, Heyuan Yan, Shi Luo, Hao Ao, Tenglong Li, Huihui Ren, Hongbin Yi, Bairen Qin, Yujia An, Bo Liu, Libin Shi, Guang Artificial Intelligence We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments. |
| title | Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2511.08892 |