Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2506.08681 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909645720780800 |
|---|---|
| author | Nguyen, Phuc Minh Nguyen, Ngoc-Hieu Nguyen, Duy H. M. Liu, Anji Mai, An Nguyen, Binh T. Sonntag, Daniel Doan, Khoa D. |
| author_facet | Nguyen, Phuc Minh Nguyen, Ngoc-Hieu Nguyen, Duy H. M. Liu, Anji Mai, An Nguyen, Binh T. Sonntag, Daniel Doan, Khoa D. |
| contents | Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) with human values. However, these methods are more susceptible to over-optimization, in which the model drifts away from the reference policy, leading to degraded performance as training progresses. This paper proposes a novel importance-sampling approach to mitigate the over-optimization problem of offline DAAs. This approach, called (IS-DAAs), multiplies the DAA objective with an importance ratio that accounts for the reference policy distribution. IS-DAAs additionally avoid the high variance issue associated with importance sampling by clipping the importance ratio to a maximum value. Our extensive experiments demonstrate that IS-DAAs can effectively mitigate over-optimization, especially under low regularization strength, and achieve better performance than other methods designed to address this problem. Our implementations are provided publicly at this link. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08681 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling Nguyen, Phuc Minh Nguyen, Ngoc-Hieu Nguyen, Duy H. M. Liu, Anji Mai, An Nguyen, Binh T. Sonntag, Daniel Doan, Khoa D. Machine Learning Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) with human values. However, these methods are more susceptible to over-optimization, in which the model drifts away from the reference policy, leading to degraded performance as training progresses. This paper proposes a novel importance-sampling approach to mitigate the over-optimization problem of offline DAAs. This approach, called (IS-DAAs), multiplies the DAA objective with an importance ratio that accounts for the reference policy distribution. IS-DAAs additionally avoid the high variance issue associated with importance sampling by clipping the importance ratio to a maximum value. Our extensive experiments demonstrate that IS-DAAs can effectively mitigate over-optimization, especially under low regularization strength, and achieve better performance than other methods designed to address this problem. Our implementations are provided publicly at this link. |
| title | Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.08681 |