RePO: Understanding Preference Learning Through ReLU-Based Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909870365605888 |
|---|---|
| author | Wu, Junkang Huang, Kexin Wang, Xue Gao, Jinyang Ding, Bolin Wu, Jiancan He, Xiangnan Wang, Xiang |
| author_facet | Wu, Junkang Huang, Kexin Wang, Xue Gao, Jinyang Ding, Bolin Wu, Jiancan He, Xiangnan Wang, Xiang |
| contents | Aligning large language models (LLMs) with human preferences is critical for real-world deployment, yet existing methods like RLHF face computational and stability challenges. While DPO establishes an offline paradigm with single hyperparameter $β$, subsequent methods like SimPO reintroduce complexity through dual parameters ($β$, $γ$). We propose {ReLU-based Preference Optimization (RePO)}, a streamlined algorithm that eliminates $β$ via two advances: (1) retaining SimPO's reference-free margins but removing $β$ through gradient analysis, and (2) adopting a ReLU-based max-margin loss that naturally filters trivial pairs. Theoretically, RePO is characterized as SimPO's limiting case ($β\to \infty$), where the logistic weighting collapses to binary thresholding, forming a convex envelope of the 0-1 loss. Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO and SimPO across multiple base models, requiring only one hyperparameter to tune. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_07426 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RePO: Understanding Preference Learning Through ReLU-Based Optimization Wu, Junkang Huang, Kexin Wang, Xue Gao, Jinyang Ding, Bolin Wu, Jiancan He, Xiangnan Wang, Xiang Machine Learning Artificial Intelligence Aligning large language models (LLMs) with human preferences is critical for real-world deployment, yet existing methods like RLHF face computational and stability challenges. While DPO establishes an offline paradigm with single hyperparameter $β$, subsequent methods like SimPO reintroduce complexity through dual parameters ($β$, $γ$). We propose {ReLU-based Preference Optimization (RePO)}, a streamlined algorithm that eliminates $β$ via two advances: (1) retaining SimPO's reference-free margins but removing $β$ through gradient analysis, and (2) adopting a ReLU-based max-margin loss that naturally filters trivial pairs. Theoretically, RePO is characterized as SimPO's limiting case ($β\to \infty$), where the logistic weighting collapses to binary thresholding, forming a convex envelope of the 0-1 loss. Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO and SimPO across multiple base models, requiring only one hyperparameter to tune. |
| title | RePO: Understanding Preference Learning Through ReLU-Based Optimization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2503.07426 |