Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917968673243136 |
|---|---|
| author | Chen, Chaoran Yao, Bingsheng Zou, Ruishi Hua, Wenyue Lyu, Weimin Ye, Yanfang Li, Toby Jia-Jun Wang, Dakuo |
| author_facet | Chen, Chaoran Yao, Bingsheng Zou, Ruishi Hua, Wenyue Lyu, Weimin Ye, Yanfang Li, Toby Jia-Jun Wang, Dakuo |
| contents | Role-Playing Agent (RPA) is an increasingly popular type of LLM Agent that simulates human-like behaviors in a variety of tasks. However, evaluating RPAs is challenging due to diverse task requirements and agent designs. This paper proposes an evidence-based, actionable, and generalizable evaluation design guideline for LLM-based RPA by systematically reviewing 1,676 papers published between Jan. 2021 and Dec. 2024. Our analysis identifies six agent attributes, seven task attributes, and seven evaluation metrics from existing literature. Based on these findings, we present an RPA evaluation design guideline to help researchers develop more systematic and consistent evaluation methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13012 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents Chen, Chaoran Yao, Bingsheng Zou, Ruishi Hua, Wenyue Lyu, Weimin Ye, Yanfang Li, Toby Jia-Jun Wang, Dakuo Human-Computer Interaction Computation and Language Role-Playing Agent (RPA) is an increasingly popular type of LLM Agent that simulates human-like behaviors in a variety of tasks. However, evaluating RPAs is challenging due to diverse task requirements and agent designs. This paper proposes an evidence-based, actionable, and generalizable evaluation design guideline for LLM-based RPA by systematically reviewing 1,676 papers published between Jan. 2021 and Dec. 2024. Our analysis identifies six agent attributes, seven task attributes, and seven evaluation metrics from existing literature. Based on these findings, we present an RPA evaluation design guideline to help researchers develop more systematic and consistent evaluation methods. |
| title | Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents |
| topic | Human-Computer Interaction Computation and Language |
| url | https://arxiv.org/abs/2502.13012 |