Larger or Smaller Reward Margins to Select Preferences for Alignment?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910856899461120 |
|---|---|
| author | Huang, Kexin Wu, Junkang Chen, Ziqian Wang, Xue Gao, Jinyang Ding, Bolin Wu, Jiancan He, Xiangnan Wang, Xiang |
| author_facet | Huang, Kexin Wu, Junkang Chen, Ziqian Wang, Xue Gao, Jinyang Ding, Bolin Wu, Jiancan He, Xiangnan Wang, Xiang |
| contents | Preference learning is critical for aligning large language models (LLMs) with human values, with the quality of preference datasets playing a crucial role in this process. While existing metrics primarily assess data quality based on either explicit or implicit reward margins, they often provide contradictory evaluations for the same data. To address this issue, we introduce the alignment potential metric, which quantifies the gap from the model's current implicit reward margin to the target explicit reward margin, thereby estimating the model's potential to align with the preference data. Empirical results demonstrate that training on data selected by this metric consistently enhances alignment performance, surpassing existing metrics across different base models and optimization objectives. Furthermore, our method extends to self-play data generation frameworks, where the metric is used to identify high-quality data within the self-generated content by LLMs. Under this data generation scenario, our method surpasses current state-of-the-art (SOTA) results across various training settings and demonstrates continuous improvements in alignment performance as dataset size and training iterations increase. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01864 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Larger or Smaller Reward Margins to Select Preferences for Alignment? Huang, Kexin Wu, Junkang Chen, Ziqian Wang, Xue Gao, Jinyang Ding, Bolin Wu, Jiancan He, Xiangnan Wang, Xiang Machine Learning Artificial Intelligence Computation and Language Preference learning is critical for aligning large language models (LLMs) with human values, with the quality of preference datasets playing a crucial role in this process. While existing metrics primarily assess data quality based on either explicit or implicit reward margins, they often provide contradictory evaluations for the same data. To address this issue, we introduce the alignment potential metric, which quantifies the gap from the model's current implicit reward margin to the target explicit reward margin, thereby estimating the model's potential to align with the preference data. Empirical results demonstrate that training on data selected by this metric consistently enhances alignment performance, surpassing existing metrics across different base models and optimization objectives. Furthermore, our method extends to self-play data generation frameworks, where the metric is used to identify high-quality data within the self-generated content by LLMs. Under this data generation scenario, our method surpasses current state-of-the-art (SOTA) results across various training settings and demonstrates continuous improvements in alignment performance as dataset size and training iterations increase. |
| title | Larger or Smaller Reward Margins to Select Preferences for Alignment? |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2503.01864 |