RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914108689874944 |
|---|---|
| author | Xiang, Hao Tang, Tianyi Su, Yang Yu, Bowen Yang, An Huang, Fei Zhang, Yichang Lu, Yaojie Lin, Hongyu Han, Xianpei Zhou, Jingren Lin, Junyang Sun, Le |
| author_facet | Xiang, Hao Tang, Tianyi Su, Yang Yu, Bowen Yang, An Huang, Fei Zhang, Yichang Lu, Yaojie Lin, Hongyu Han, Xianpei Zhou, Jingren Lin, Junyang Sun, Le |
| contents | Recent advancements in Large Language Models (LLMs) have shown outstanding potential for role-playing applications. Evaluating these capabilities is becoming crucial yet remains challenging. Existing benchmarks mostly adopt a \textbf{character-centric} approach, simplify user-character interactions to isolated Q&A tasks, and fail to reflect real-world applications. To address this limitation, we introduce RMTBench, a comprehensive \textbf{user-centric} bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. RMTBench includes custom characters with detailed backgrounds and abstract characters defined by simple traits, enabling evaluation across various user scenarios. Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications. Furthermore, we construct an authentic multi-turn dialogue simulation mechanism. With carefully selected evaluation dimensions and LLM-based scoring, this mechanism captures the complex intention of conversations between the user and the character. By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements, offering a more effective framework for assessing role-playing capabilities in LLMs. All code and datasets will be released soon. We release the datasets at https://huggingface.co/datasets/xiangh/RMTBENCH. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_20352 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing Xiang, Hao Tang, Tianyi Su, Yang Yu, Bowen Yang, An Huang, Fei Zhang, Yichang Lu, Yaojie Lin, Hongyu Han, Xianpei Zhou, Jingren Lin, Junyang Sun, Le Computation and Language Recent advancements in Large Language Models (LLMs) have shown outstanding potential for role-playing applications. Evaluating these capabilities is becoming crucial yet remains challenging. Existing benchmarks mostly adopt a \textbf{character-centric} approach, simplify user-character interactions to isolated Q&A tasks, and fail to reflect real-world applications. To address this limitation, we introduce RMTBench, a comprehensive \textbf{user-centric} bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. RMTBench includes custom characters with detailed backgrounds and abstract characters defined by simple traits, enabling evaluation across various user scenarios. Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications. Furthermore, we construct an authentic multi-turn dialogue simulation mechanism. With carefully selected evaluation dimensions and LLM-based scoring, this mechanism captures the complex intention of conversations between the user and the character. By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements, offering a more effective framework for assessing role-playing capabilities in LLMs. All code and datasets will be released soon. We release the datasets at https://huggingface.co/datasets/xiangh/RMTBENCH. |
| title | RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.20352 |