K-order Ranking Preference Optimization for Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912405951348736 |
|---|---|
| author | Cai, Shihao Gao, Chongming Zhang, Yang Shi, Wentao Zhang, Jizhi Bao, Keqin Wang, Qifan Feng, Fuli |
| author_facet | Cai, Shihao Gao, Chongming Zhang, Yang Shi, Wentao Zhang, Jizhi Bao, Keqin Wang, Qifan Feng, Fuli |
| contents | To adapt large language models (LLMs) to ranking tasks, existing list-wise methods, represented by list-wise Direct Preference Optimization (DPO), focus on optimizing partial-order or full-order list ranking consistency for LLMs to enhance their ranking abilities. However, we argue that optimizing top-K ranking consistency could be more appropriate for real-world applications. There are two main reasons: (1) users are typically concerned with only the top-K results, making top-K ranking more important, and (2) tail items often lack precise feedback, making top-K ranking more reliable. Based on this, we propose K-order Ranking Preference Optimization (KPO) by extending the DPO's Plackett-Luce model to accommodate top-K rankings. Additionally, recognizing that the number of important items can vary across queries, we extend KPO to dynamically determine appropriate K for different samples and introduce a curriculum learning strategy to boost training efficiency. Extensive experiments demonstrate the effectiveness of KPO, highlighting its high sample efficiency and robustness to noise. The code is available at https://github.com/Lanyu0303/KPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00441 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | K-order Ranking Preference Optimization for Large Language Models Cai, Shihao Gao, Chongming Zhang, Yang Shi, Wentao Zhang, Jizhi Bao, Keqin Wang, Qifan Feng, Fuli Information Retrieval To adapt large language models (LLMs) to ranking tasks, existing list-wise methods, represented by list-wise Direct Preference Optimization (DPO), focus on optimizing partial-order or full-order list ranking consistency for LLMs to enhance their ranking abilities. However, we argue that optimizing top-K ranking consistency could be more appropriate for real-world applications. There are two main reasons: (1) users are typically concerned with only the top-K results, making top-K ranking more important, and (2) tail items often lack precise feedback, making top-K ranking more reliable. Based on this, we propose K-order Ranking Preference Optimization (KPO) by extending the DPO's Plackett-Luce model to accommodate top-K rankings. Additionally, recognizing that the number of important items can vary across queries, we extend KPO to dynamically determine appropriate K for different samples and introduce a curriculum learning strategy to boost training efficiency. Extensive experiments demonstrate the effectiveness of KPO, highlighting its high sample efficiency and robustness to noise. The code is available at https://github.com/Lanyu0303/KPO. |
| title | K-order Ranking Preference Optimization for Large Language Models |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2506.00441 |