Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910901476524032 |
|---|---|
| author | Chifu, Adrian-Gabriel Déjean, Sébastien Mothe, Josiane Garouani, Moncef Ortiz, Diego Ullah, Md Zia |
| author_facet | Chifu, Adrian-Gabriel Déjean, Sébastien Mothe, Josiane Garouani, Moncef Ortiz, Diego Ullah, Md Zia |
| contents | Query Performance Prediction (QPP) estimates retrieval systems effectiveness for a given query, offering valuable insights for search effectiveness and query processing. Despite extensive research, QPPs face critical challenges in generalizing across diverse retrieval paradigms and collections. This paper provides a comprehensive evaluation of state-of-the-art QPPs (e.g. NQC, UQC), LETOR-based features, and newly explored dense-based predictors. Using diverse sparse rankers (BM25, DFree without and with query expansion) and hybrid or dense (SPLADE and ColBert) rankers and diverse test collections ROBUST, GOV2, WT10G, and MS MARCO; we investigate the relationships between predicted and actual performance, with a focus on generalization and robustness. Results show significant variability in predictors accuracy, with collections as the main factor and rankers next. Some sparse predictors perform somehow on some collections (TREC ROBUST and GOV2) but do not generalise to other collections (WT10G and MS-MARCO). While some predictors show promise in specific scenarios, their overall limitations constrain their utility for applications. We show that QPP-driven selective query processing offers only marginal gains, emphasizing the need for improved predictors that generalize across collections, align with dense retrieval architectures and are useful for downstream applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01101 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing Chifu, Adrian-Gabriel Déjean, Sébastien Mothe, Josiane Garouani, Moncef Ortiz, Diego Ullah, Md Zia Information Retrieval Machine Learning H.3; H.3.3; I.2; I.2.7 Query Performance Prediction (QPP) estimates retrieval systems effectiveness for a given query, offering valuable insights for search effectiveness and query processing. Despite extensive research, QPPs face critical challenges in generalizing across diverse retrieval paradigms and collections. This paper provides a comprehensive evaluation of state-of-the-art QPPs (e.g. NQC, UQC), LETOR-based features, and newly explored dense-based predictors. Using diverse sparse rankers (BM25, DFree without and with query expansion) and hybrid or dense (SPLADE and ColBert) rankers and diverse test collections ROBUST, GOV2, WT10G, and MS MARCO; we investigate the relationships between predicted and actual performance, with a focus on generalization and robustness. Results show significant variability in predictors accuracy, with collections as the main factor and rankers next. Some sparse predictors perform somehow on some collections (TREC ROBUST and GOV2) but do not generalise to other collections (WT10G and MS-MARCO). While some predictors show promise in specific scenarios, their overall limitations constrain their utility for applications. We show that QPP-driven selective query processing offers only marginal gains, emphasizing the need for improved predictors that generalize across collections, align with dense retrieval architectures and are useful for downstream applications. |
| title | Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing |
| topic | Information Retrieval Machine Learning H.3; H.3.3; I.2; I.2.7 |
| url | https://arxiv.org/abs/2504.01101 |