RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913674076094464 |
|---|---|
| author | Yu, Pengfei Shen, Dongming Meng, Silin Lee, Jaewon Yin, Weisu Cui, Andrea Yaoyun Xu, Zhenlin Zhu, Yi Shi, Xingjian Li, Mu Smola, Alex |
| author_facet | Yu, Pengfei Shen, Dongming Meng, Silin Lee, Jaewon Yin, Weisu Cui, Andrea Yaoyun Xu, Zhenlin Zhu, Yi Shi, Xingjian Li, Mu Smola, Alex |
| contents | We present RPGBench, the first benchmark designed to evaluate large language models (LLMs) as text-based role-playing game (RPG) engines. RPGBench comprises two core tasks: Game Creation (GC) and Game Simulation (GS). In GC, an LLM must craft a valid and playable RPG world using a structured event-state representation, ensuring logical coherence and proper termination conditions. In GS, the LLM simulates interactive gameplay across multiple rounds while consistently updating states and enforcing game rules. To comprehensively assess performance, RPGBench integrates objective and subjective evaluation methodologies. Objective measures verify adherence to event mechanics and check variable updates without requiring human intervention. Subjective measures, such as content interestingness, action quality, and role-playing capability, are evaluated via an LLM-as-a-judge framework, where a strong LLM grades each candidate's outputs. Empirical results demonstrate that state-of-the-art LLMs can produce engaging stories but often struggle to implement consistent, verifiable game mechanics, particularly in long or complex scenarios. By combining structured, rule-based assessments with LLM-based judgments, RPGBench provides a new standard for evaluating how well LLMs can balance creativity, coherence, and complexity in text-based RPGs, opening avenues for more immersive and controllable interactive storytelling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00595 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines Yu, Pengfei Shen, Dongming Meng, Silin Lee, Jaewon Yin, Weisu Cui, Andrea Yaoyun Xu, Zhenlin Zhu, Yi Shi, Xingjian Li, Mu Smola, Alex Computation and Language Artificial Intelligence We present RPGBench, the first benchmark designed to evaluate large language models (LLMs) as text-based role-playing game (RPG) engines. RPGBench comprises two core tasks: Game Creation (GC) and Game Simulation (GS). In GC, an LLM must craft a valid and playable RPG world using a structured event-state representation, ensuring logical coherence and proper termination conditions. In GS, the LLM simulates interactive gameplay across multiple rounds while consistently updating states and enforcing game rules. To comprehensively assess performance, RPGBench integrates objective and subjective evaluation methodologies. Objective measures verify adherence to event mechanics and check variable updates without requiring human intervention. Subjective measures, such as content interestingness, action quality, and role-playing capability, are evaluated via an LLM-as-a-judge framework, where a strong LLM grades each candidate's outputs. Empirical results demonstrate that state-of-the-art LLMs can produce engaging stories but often struggle to implement consistent, verifiable game mechanics, particularly in long or complex scenarios. By combining structured, rule-based assessments with LLM-based judgments, RPGBench provides a new standard for evaluating how well LLMs can balance creativity, coherence, and complexity in text-based RPGs, opening avenues for more immersive and controllable interactive storytelling. |
| title | RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.00595 |