Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909534015979520 |
|---|---|
| author | Xu, Rui Wang, MingYu Wang, XinTao Lu, Dakuan Tan, Xiaoyu Chu, Wei Xu, Yinghui |
| author_facet | Xu, Rui Wang, MingYu Wang, XinTao Lu, Dakuan Tan, Xiaoyu Chu, Wei Xu, Yinghui |
| contents | Recent advances in LLM-based role-playing language agents (RPLAs) have attracted broad attention in various applications. While chain-of-thought reasoning has shown importance in many tasks for LLMs, the internal thinking processes of RPLAs remain unexplored. Understanding characters' inner thoughts is crucial for developing advanced RPLAs. In this paper, we introduce ROLETHINK, a novel benchmark constructed from literature for evaluating character thought generation. We propose the task of inner thought reasoning, which includes two sets: the gold set that compares generated thoughts with original character monologues, and the silver set that uses expert synthesized character analyses as references. To address this challenge, we propose MIRROR, a chain-of-thought approach that generates character thoughts by retrieving memories, predicting character reactions, and synthesizing motivations. Through extensive experiments, we demonstrate the importance of inner thought reasoning for RPLAs, and MIRROR consistently outperforms existing methods. Resources are available at https://github.com/airaer1998/RPA_Thought. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_08193 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents Xu, Rui Wang, MingYu Wang, XinTao Lu, Dakuan Tan, Xiaoyu Chu, Wei Xu, Yinghui Artificial Intelligence Recent advances in LLM-based role-playing language agents (RPLAs) have attracted broad attention in various applications. While chain-of-thought reasoning has shown importance in many tasks for LLMs, the internal thinking processes of RPLAs remain unexplored. Understanding characters' inner thoughts is crucial for developing advanced RPLAs. In this paper, we introduce ROLETHINK, a novel benchmark constructed from literature for evaluating character thought generation. We propose the task of inner thought reasoning, which includes two sets: the gold set that compares generated thoughts with original character monologues, and the silver set that uses expert synthesized character analyses as references. To address this challenge, we propose MIRROR, a chain-of-thought approach that generates character thoughts by retrieving memories, predicting character reactions, and synthesizing motivations. Through extensive experiments, we demonstrate the importance of inner thought reasoning for RPLAs, and MIRROR consistently outperforms existing methods. Resources are available at https://github.com/airaer1998/RPA_Thought. |
| title | Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2503.08193 |