Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.06036 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914539340038144 |
|---|---|
| author | Pan, Licheng Yang, Haochen Li, Haoxuan Lu, Yunsheng Tong, Yongqi Wang, Yinuo Wang, Shijian Chu, Zhixuan Shen, Lei Lu, Yuan Wang, Hao |
| author_facet | Pan, Licheng Yang, Haochen Li, Haoxuan Lu, Yunsheng Tong, Yongqi Wang, Yinuo Wang, Shijian Chu, Zhixuan Shen, Lei Lu, Yuan Wang, Hao |
| contents | Reward models are fundamental to Reinforcement Learning from Human Feedback (RLHF), yet real-world datasets are inevitably corrupted by noisy preference. Conventional training objectives tend to overfit these errors, while existing denoising approaches often rely on homogeneous noise assumptions that fail to capture the complexity of linguistic preferences. To handle these challenges, we propose SelectiveRM, a framework grounded in optimal transport. We first devise a Joint Consistency Discrepancy to align the distribution of model predictions with preference data. Furthermore, to address the limitation of strict mass conservation which compels the model to fit outliers, we incorporate a Mass Relaxation mechanism via partial transport. This enables the autonomous exclusion of samples with noisy preference that contradict semantic consistency. Theoretically, we demonstrate that SelectiveRM optimizes a tighter upper bound on the unobserved clean risk. Extensive experiments validate that our approach significantly outperforms state-of-the-art baselines across diverse benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06036 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Optimal Transport for LLM Reward Modeling from Noisy Preference Pan, Licheng Yang, Haochen Li, Haoxuan Lu, Yunsheng Tong, Yongqi Wang, Yinuo Wang, Shijian Chu, Zhixuan Shen, Lei Lu, Yuan Wang, Hao Machine Learning Artificial Intelligence Reward models are fundamental to Reinforcement Learning from Human Feedback (RLHF), yet real-world datasets are inevitably corrupted by noisy preference. Conventional training objectives tend to overfit these errors, while existing denoising approaches often rely on homogeneous noise assumptions that fail to capture the complexity of linguistic preferences. To handle these challenges, we propose SelectiveRM, a framework grounded in optimal transport. We first devise a Joint Consistency Discrepancy to align the distribution of model predictions with preference data. Furthermore, to address the limitation of strict mass conservation which compels the model to fit outliers, we incorporate a Mass Relaxation mechanism via partial transport. This enables the autonomous exclusion of samples with noisy preference that contradict semantic consistency. Theoretically, we demonstrate that SelectiveRM optimizes a tighter upper bound on the unobserved clean risk. Extensive experiments validate that our approach significantly outperforms state-of-the-art baselines across diverse benchmarks. |
| title | Optimal Transport for LLM Reward Modeling from Noisy Preference |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.06036 |