Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.23923 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912810050519040 |
|---|---|
| author | Fang, Feiteng Chen, Dingwei Huang, Xiang Lin, Ting-En Wu, Yuchuan Liu, Xiong Ye, Xinge Liu, Ziqiang Zhang, Haonan Zhu, Liang Alinejad-Rokny, Hamid Yang, Min Li, Yongbin |
| author_facet | Fang, Feiteng Chen, Dingwei Huang, Xiang Lin, Ting-En Wu, Yuchuan Liu, Xiong Ye, Xinge Liu, Ziqiang Zhang, Haonan Zhu, Liang Alinejad-Rokny, Hamid Yang, Min Li, Yongbin |
| contents | Currently, most reinforcement learning tasks focus on domains like mathematics and programming, where verification is relatively straightforward. However, in subjective tasks such as role-playing, alignment techniques struggle to make progress, primarily because subjective reward modeling using the Bradley-Terry model faces significant challenges when dealing with ambiguous preferences. To improve reward modeling in subjective tasks, this paper proposes AAM (\textbf{\underline{A}}ct-\textbf{\underline{A}}daptive \textbf{\underline{M}}argin), which enhances reward modeling by dynamically calibrating preference margins using the model's internal parameter knowledge. We design two versions of AAM that efficiently generate contextually-appropriate preference gaps without additional human annotation. This approach fundamentally improves how reward models handle subjective rewards by better integrating generative understanding with preference scoring. To validate AAM's effectiveness in subjective reward modeling, we conduct evaluations on RewardBench, JudgeBench, and challenging role-playing tasks. Results show that AAM significantly improves subjective reward modeling performance, enhancing Bradley-Terry reward models by 2.95\% in general tasks and 4.85\% in subjective role-playing tasks. Furthermore, reward models trained with AAM can help downstream alignment tasks achieve better results. Our test results show that applying rewards generated by AAM-Augmented RM to preference learning techniques (e.g., GRPO) achieves state-of-the-art results on CharacterEval and Charm. Code and dataset are available at https://github.com/calubkk/AAM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23923 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity Fang, Feiteng Chen, Dingwei Huang, Xiang Lin, Ting-En Wu, Yuchuan Liu, Xiong Ye, Xinge Liu, Ziqiang Zhang, Haonan Zhu, Liang Alinejad-Rokny, Hamid Yang, Min Li, Yongbin Computation and Language Artificial Intelligence Currently, most reinforcement learning tasks focus on domains like mathematics and programming, where verification is relatively straightforward. However, in subjective tasks such as role-playing, alignment techniques struggle to make progress, primarily because subjective reward modeling using the Bradley-Terry model faces significant challenges when dealing with ambiguous preferences. To improve reward modeling in subjective tasks, this paper proposes AAM (\textbf{\underline{A}}ct-\textbf{\underline{A}}daptive \textbf{\underline{M}}argin), which enhances reward modeling by dynamically calibrating preference margins using the model's internal parameter knowledge. We design two versions of AAM that efficiently generate contextually-appropriate preference gaps without additional human annotation. This approach fundamentally improves how reward models handle subjective rewards by better integrating generative understanding with preference scoring. To validate AAM's effectiveness in subjective reward modeling, we conduct evaluations on RewardBench, JudgeBench, and challenging role-playing tasks. Results show that AAM significantly improves subjective reward modeling performance, enhancing Bradley-Terry reward models by 2.95\% in general tasks and 4.85\% in subjective role-playing tasks. Furthermore, reward models trained with AAM can help downstream alignment tasks achieve better results. Our test results show that applying rewards generated by AAM-Augmented RM to preference learning techniques (e.g., GRPO) achieves state-of-the-art results on CharacterEval and Charm. Code and dataset are available at https://github.com/calubkk/AAM. |
| title | Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.23923 |