GameDevBench: Evaluating Agentic Capabilities Through Game Development
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915792869654528 |
|---|---|
| author | Chi, Wayne Fang, Yixiong Yayavaram, Arnav Yayavaram, Siddharth Karten, Seth Wei, Qiuhong Anna Chen, Runkun Wang, Alexander Chen, Valerie Talwalkar, Ameet Donahue, Chris |
| author_facet | Chi, Wayne Fang, Yixiong Yayavaram, Arnav Yayavaram, Siddharth Karten, Seth Wei, Qiuhong Anna Chen, Runkun Wang, Alexander Chen, Valerie Talwalkar, Ameet Donahue, Chris |
| contents | Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. Game development provides such a testbed as agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 132 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex -- the average solution requires over three times the amount of lines of code and file changes compared to prior software development benchmarks. Agents still struggle with game development, with the best agent solving only 54.5% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with success rates dropping from 46.9% on gameplay-oriented tasks to 31.6% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, with the largest change being an increase in Claude Sonnet 4.5's performance from 33.3% to 47.7%. We release GameDevBench publicly to support further research into agentic game development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11103 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GameDevBench: Evaluating Agentic Capabilities Through Game Development Chi, Wayne Fang, Yixiong Yayavaram, Arnav Yayavaram, Siddharth Karten, Seth Wei, Qiuhong Anna Chen, Runkun Wang, Alexander Chen, Valerie Talwalkar, Ameet Donahue, Chris Artificial Intelligence Computation and Language Software Engineering Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. Game development provides such a testbed as agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 132 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex -- the average solution requires over three times the amount of lines of code and file changes compared to prior software development benchmarks. Agents still struggle with game development, with the best agent solving only 54.5% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with success rates dropping from 46.9% on gameplay-oriented tasks to 31.6% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, with the largest change being an increase in Claude Sonnet 4.5's performance from 33.3% to 47.7%. We release GameDevBench publicly to support further research into agentic game development. |
| title | GameDevBench: Evaluating Agentic Capabilities Through Game Development |
| topic | Artificial Intelligence Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2602.11103 |