NARRA-Gym for Evaluating Interactive Narrative Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917475006808064 |
|---|---|
| author | Huang, Yue Ma, Yuchen Ye, Jiayi Wang, Wenjie Ling, Zipeng Hu, Xingjian Hao, Yuexing Chen, Zichen Xu, Zhangchen He, Yunhong Yuan, Zhengqing Zhou, Yujun Guo, Kehan Chen, Chaoran Li, Toby Jia-Jun Feuerriegel, Stefan Zhang, Xiangliang |
| author_facet | Huang, Yue Ma, Yuchen Ye, Jiayi Wang, Wenjie Ling, Zipeng Hu, Xingjian Hao, Yuexing Chen, Zichen Xu, Zhangchen He, Yunhong Yuan, Zhengqing Zhou, Yujun Guo, Kehan Chen, Chaoran Li, Toby Jia-Jun Feuerriegel, Stefan Zhang, Xiangliang |
| contents | Interactive narrative tasks require LLMs to sustain a coherent, evolving story while adapting to a user over multiple turns. However, suitable benchmarks for this setting are limited: existing evaluations often focus on static prompts, isolated story generations, or post-hoc ratings, and therefore miss whether models can jointly manage story generation, long-context state and pacing, character simulation, empathic personalization, and story-grounded artifacts. We introduce NARRA-Gym, an executable evaluation environment that turns a sparse emotional seed into a complete interactive story episode and logs the full model-in-the-loop trajectory, including story construction, memory updates, planning, pacing interventions, and optional artifact synthesis. We evaluate nine frontier LLMs using a controlled LLM-as-judge sweep over eight benchmark personas and a human evaluation in which participants rate customized model outputs. Our results show substantial variation across models, personas, and evaluation dimensions: models that produce fluent stories can still fail on robustness, user experience, or resistance-sensitive personalization. These findings suggest that interactive narrative offers a useful benchmark for evaluating long-horizon, user-adaptive LLM behavior beyond isolated story quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_08503 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | NARRA-Gym for Evaluating Interactive Narrative Agents Huang, Yue Ma, Yuchen Ye, Jiayi Wang, Wenjie Ling, Zipeng Hu, Xingjian Hao, Yuexing Chen, Zichen Xu, Zhangchen He, Yunhong Yuan, Zhengqing Zhou, Yujun Guo, Kehan Chen, Chaoran Li, Toby Jia-Jun Feuerriegel, Stefan Zhang, Xiangliang Computation and Language Computers and Society Human-Computer Interaction Interactive narrative tasks require LLMs to sustain a coherent, evolving story while adapting to a user over multiple turns. However, suitable benchmarks for this setting are limited: existing evaluations often focus on static prompts, isolated story generations, or post-hoc ratings, and therefore miss whether models can jointly manage story generation, long-context state and pacing, character simulation, empathic personalization, and story-grounded artifacts. We introduce NARRA-Gym, an executable evaluation environment that turns a sparse emotional seed into a complete interactive story episode and logs the full model-in-the-loop trajectory, including story construction, memory updates, planning, pacing interventions, and optional artifact synthesis. We evaluate nine frontier LLMs using a controlled LLM-as-judge sweep over eight benchmark personas and a human evaluation in which participants rate customized model outputs. Our results show substantial variation across models, personas, and evaluation dimensions: models that produce fluent stories can still fail on robustness, user experience, or resistance-sensitive personalization. These findings suggest that interactive narrative offers a useful benchmark for evaluating long-horizon, user-adaptive LLM behavior beyond isolated story quality. |
| title | NARRA-Gym for Evaluating Interactive Narrative Agents |
| topic | Computation and Language Computers and Society Human-Computer Interaction |
| url | https://arxiv.org/abs/2605.08503 |