Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915497075802112 |
|---|---|
| author | Janiak, Denis Moska, Julia Motyka, Dawid Seweryn, Karolina Walkowiak, Paweł Żuk, Bartosz Janz, Arkadiusz |
| author_facet | Janiak, Denis Moska, Julia Motyka, Dawid Seweryn, Karolina Walkowiak, Paweł Żuk, Bartosz Janz, Arkadiusz |
| contents | Large language models (LLMs) require careful alignment to balance competing objectives - factuality, safety, conciseness, proactivity, and diversity. Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. We propose a unified evaluation framework that compares LLM alignment methods (PPO, DPO, ORPO, KTO) across these five axes, using both in-distribution and out-of-distribution datasets. Leveraging a specialized LLM-as-Judge prompt, validated through human studies, we reveal that DPO and KTO excel in factual accuracy, PPO and DPO lead in safety, and PPO best balances conciseness with proactivity. Our findings provide insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12936 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety Janiak, Denis Moska, Julia Motyka, Dawid Seweryn, Karolina Walkowiak, Paweł Żuk, Bartosz Janz, Arkadiusz Machine Learning Computation and Language Large language models (LLMs) require careful alignment to balance competing objectives - factuality, safety, conciseness, proactivity, and diversity. Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. We propose a unified evaluation framework that compares LLM alignment methods (PPO, DPO, ORPO, KTO) across these five axes, using both in-distribution and out-of-distribution datasets. Leveraging a specialized LLM-as-Judge prompt, validated through human studies, we reveal that DPO and KTO excel in factual accuracy, PPO and DPO lead in safety, and PPO best balances conciseness with proactivity. Our findings provide insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs. |
| title | Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2509.12936 |