Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916830053924864 |
|---|---|
| author | Wu, Junru Yan, Le Qin, Zhen Zhuang, Honglei C., Paul Suganthan G. Liu, Tianqi Dong, Zhe Wang, Xuanhui Oosterhuis, Harrie |
| author_facet | Wu, Junru Yan, Le Qin, Zhen Zhuang, Honglei C., Paul Suganthan G. Liu, Tianqi Dong, Zhe Wang, Xuanhui Oosterhuis, Harrie |
| contents | While Pairwise Ranking Prompting (PRP) with Large Language Models (LLMs) is one of the most effective zero-shot document ranking methods, it has a quadratic computational complexity with respect to the number of documents to be ranked, as it requires an enumeration over all possible document pairs. Consequently, the outstanding ranking performance of PRP has remained unreachable for most real-world ranking applications.
In this work, we propose to harness the effectiveness of PRP through pairwise distillation. Specifically, we distill a pointwise student ranker from pairwise teacher labels generated by PRP, resulting in an efficient student model that retains the performance of PRP with substantially lower computational costs. Furthermore, we find that the distillation process can be made sample-efficient: with only 2% of pairs, we are able to obtain the same performance as using all pairs for teacher labels. Thus, our novel approach provides a solution to harness the ranking performance of PRP without incurring high computational costs during both distillation and serving. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04820 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation Wu, Junru Yan, Le Qin, Zhen Zhuang, Honglei C., Paul Suganthan G. Liu, Tianqi Dong, Zhe Wang, Xuanhui Oosterhuis, Harrie Information Retrieval While Pairwise Ranking Prompting (PRP) with Large Language Models (LLMs) is one of the most effective zero-shot document ranking methods, it has a quadratic computational complexity with respect to the number of documents to be ranked, as it requires an enumeration over all possible document pairs. Consequently, the outstanding ranking performance of PRP has remained unreachable for most real-world ranking applications. In this work, we propose to harness the effectiveness of PRP through pairwise distillation. Specifically, we distill a pointwise student ranker from pairwise teacher labels generated by PRP, resulting in an efficient student model that retains the performance of PRP with substantially lower computational costs. Furthermore, we find that the distillation process can be made sample-efficient: with only 2% of pairs, we are able to obtain the same performance as using all pairs for teacher labels. Thus, our novel approach provides a solution to harness the ranking performance of PRP without incurring high computational costs during both distillation and serving. |
| title | Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2507.04820 |