Learning to play: A Multimodal Agent for 3D Game-Play
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908602033242112 |
|---|---|
| author | Yue, Yuguang Salia, Irakli Hunt, Samuel Green, Christopher Shi, Wenzhe Hunt, Jonathan J |
| author_facet | Yue, Yuguang Salia, Irakli Hunt, Samuel Green, Christopher Shi, Wenzhe Hunt, Jonathan J |
| contents | We argue that 3-D first-person video games are a challenging environment for real-time multi-modal reasoning. We first describe our dataset of human game-play, collected across a large variety of 3-D first-person games, which is both substantially larger and more diverse compared to prior publicly disclosed datasets, and contains text instructions. We demonstrate that we can learn an inverse dynamics model from this dataset, which allows us to impute actions on a much larger dataset of publicly available videos of human game play that lack recorded actions. We then train a text-conditioned agent for game playing using behavior cloning, with a custom architecture capable of realtime inference on a consumer GPU. We show the resulting model is capable of playing a variety of 3-D games and responding to text input. Finally, we outline some of the remaining challenges such as long-horizon tasks and quantitative evaluation across a large set of games. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_16774 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning to play: A Multimodal Agent for 3D Game-Play Yue, Yuguang Salia, Irakli Hunt, Samuel Green, Christopher Shi, Wenzhe Hunt, Jonathan J Machine Learning Artificial Intelligence We argue that 3-D first-person video games are a challenging environment for real-time multi-modal reasoning. We first describe our dataset of human game-play, collected across a large variety of 3-D first-person games, which is both substantially larger and more diverse compared to prior publicly disclosed datasets, and contains text instructions. We demonstrate that we can learn an inverse dynamics model from this dataset, which allows us to impute actions on a much larger dataset of publicly available videos of human game play that lack recorded actions. We then train a text-conditioned agent for game playing using behavior cloning, with a custom architecture capable of realtime inference on a consumer GPU. We show the resulting model is capable of playing a variety of 3-D games and responding to text input. Finally, we outline some of the remaining challenges such as long-horizon tasks and quantitative evaluation across a large set of games. |
| title | Learning to play: A Multimodal Agent for 3D Game-Play |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.16774 |