Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929529225740288 |
|---|---|
| author | Sinhababu, Nilanjan Parry, Andrew Ganguly, Debasis Samanta, Debasis Mitra, Pabitra |
| author_facet | Sinhababu, Nilanjan Parry, Andrew Ganguly, Debasis Samanta, Debasis Mitra, Pabitra |
| contents | A supervised ranking model, despite its advantage of being effective, usually involves complex processing - typically multiple stages of task-specific pre-training and fine-tuning. This has motivated researchers to explore simpler pipelines leveraging large language models (LLMs) that are capable of working in a zero-shot manner. However, since zero-shot inference does not make use of a training set of pairs of queries and their relevant documents, its performance is mostly worse than that of supervised models, which are trained on such example pairs. Motivated by the existing findings that training examples generally improve zero-shot performance, in our work, we explore if this also applies to ranking models. More specifically, given a query and a pair of documents, the preference prediction task is improved by augmenting examples of preferences for similar queries from a training set. Our proposed pairwise few-shot ranker demonstrates consistent improvements over the zero-shot baseline on both in-domain (TREC DL) and out-domain (BEIR subset) retrieval benchmarks. Our method also achieves a close performance to that of a supervised model without requiring any complex training pipeline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17745 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model Sinhababu, Nilanjan Parry, Andrew Ganguly, Debasis Samanta, Debasis Mitra, Pabitra Information Retrieval Computation and Language Machine Learning A supervised ranking model, despite its advantage of being effective, usually involves complex processing - typically multiple stages of task-specific pre-training and fine-tuning. This has motivated researchers to explore simpler pipelines leveraging large language models (LLMs) that are capable of working in a zero-shot manner. However, since zero-shot inference does not make use of a training set of pairs of queries and their relevant documents, its performance is mostly worse than that of supervised models, which are trained on such example pairs. Motivated by the existing findings that training examples generally improve zero-shot performance, in our work, we explore if this also applies to ranking models. More specifically, given a query and a pair of documents, the preference prediction task is improved by augmenting examples of preferences for similar queries from a training set. Our proposed pairwise few-shot ranker demonstrates consistent improvements over the zero-shot baseline on both in-domain (TREC DL) and out-domain (BEIR subset) retrieval benchmarks. Our method also achieves a close performance to that of a supervised model without requiring any complex training pipeline. |
| title | Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model |
| topic | Information Retrieval Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2409.17745 |