RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909957022023680 |
|---|---|
| author | Ding, Hang Feng, Qiming Liu, Dongqi Zhao, Qi Yao, Tao Wang, Shuo Chen, Dongsheng Li, Jian Gan, Zhenye Zhang, Jiangning Wang, Chengjie Wang, Yabiao |
| author_facet | Ding, Hang Feng, Qiming Liu, Dongqi Zhao, Qi Yao, Tao Wang, Shuo Chen, Dongsheng Li, Jian Gan, Zhenye Zhang, Jiangning Wang, Chengjie Wang, Yabiao |
| contents | Reward modeling has become a cornerstone of aligning large language models (LLMs) with human preferences. Yet, when extended to subjective and open-ended domains such as role play, existing reward models exhibit severe degradation, struggling to capture nuanced and persona-grounded human judgments. To address this gap, we introduce RoleRMBench, the first systematic benchmark for reward modeling in role-playing dialogue, covering seven fine-grained capabilities from narrative management to role consistency and engagement. Evaluation on RoleRMBench reveals large and consistent gaps between general-purpose reward models and human judgment, particularly in narrative and stylistic dimensions. We further propose RoleRM, a reward model trained with Continuous Implicit Preferences (CIP), which reformulates subjective evaluation as continuous consistent pairwise supervision under multiple structuring strategies. Comprehensive experiments show that RoleRM surpasses strong open- and closed-source reward models by over 24% on average, demonstrating substantial gains in narrative coherence and stylistic fidelity. Our findings highlight the importance of continuous preference representation and annotation consistency, establishing a foundation for subjective alignment in human-centered dialogue systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_10575 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems Ding, Hang Feng, Qiming Liu, Dongqi Zhao, Qi Yao, Tao Wang, Shuo Chen, Dongsheng Li, Jian Gan, Zhenye Zhang, Jiangning Wang, Chengjie Wang, Yabiao Computation and Language Reward modeling has become a cornerstone of aligning large language models (LLMs) with human preferences. Yet, when extended to subjective and open-ended domains such as role play, existing reward models exhibit severe degradation, struggling to capture nuanced and persona-grounded human judgments. To address this gap, we introduce RoleRMBench, the first systematic benchmark for reward modeling in role-playing dialogue, covering seven fine-grained capabilities from narrative management to role consistency and engagement. Evaluation on RoleRMBench reveals large and consistent gaps between general-purpose reward models and human judgment, particularly in narrative and stylistic dimensions. We further propose RoleRM, a reward model trained with Continuous Implicit Preferences (CIP), which reformulates subjective evaluation as continuous consistent pairwise supervision under multiple structuring strategies. Comprehensive experiments show that RoleRM surpasses strong open- and closed-source reward models by over 24% on average, demonstrating substantial gains in narrative coherence and stylistic fidelity. Our findings highlight the importance of continuous preference representation and annotation consistency, establishing a foundation for subjective alignment in human-centered dialogue systems. |
| title | RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.10575 |