On Symmetric Losses for Robust Policy Optimization with Noisy Preferences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912404271529984 |
|---|---|
| author | Nishimori, Soichiro Zhang, Yu-Jie Lodkaew, Thanawat Sugiyama, Masashi |
| author_facet | Nishimori, Soichiro Zhang, Yu-Jie Lodkaew, Thanawat Sugiyama, Masashi |
| contents | Optimizing policies based on human preferences is key to aligning language models with human intent. This work focuses on reward modeling, a core component in reinforcement learning from human feedback (RLHF), and offline preference optimization, such as direct preference optimization. Conventional approaches typically assume accurate annotations. However, real-world preference data often contains noise due to human errors or biases. We propose a principled framework for robust policy optimization under noisy preferences, viewing reward modeling as a classification problem. This allows us to leverage symmetric losses, known for their robustness to label noise in classification, leading to our Symmetric Preference Optimization (SymPO) method. We prove that symmetric losses enable successful policy optimization even under noisy labels, as the resulting reward remains rank-preserving -- a property sufficient for policy improvement. Experiments on synthetic and real-world tasks demonstrate the effectiveness of SymPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24709 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On Symmetric Losses for Robust Policy Optimization with Noisy Preferences Nishimori, Soichiro Zhang, Yu-Jie Lodkaew, Thanawat Sugiyama, Masashi Machine Learning Artificial Intelligence Optimizing policies based on human preferences is key to aligning language models with human intent. This work focuses on reward modeling, a core component in reinforcement learning from human feedback (RLHF), and offline preference optimization, such as direct preference optimization. Conventional approaches typically assume accurate annotations. However, real-world preference data often contains noise due to human errors or biases. We propose a principled framework for robust policy optimization under noisy preferences, viewing reward modeling as a classification problem. This allows us to leverage symmetric losses, known for their robustness to label noise in classification, leading to our Symmetric Preference Optimization (SymPO) method. We prove that symmetric losses enable successful policy optimization even under noisy labels, as the resulting reward remains rank-preserving -- a property sufficient for policy improvement. Experiments on synthetic and real-world tasks demonstrate the effectiveness of SymPO. |
| title | On Symmetric Losses for Robust Policy Optimization with Noisy Preferences |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.24709 |