Cover Image

Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Otero, David, Parapar, Javier, Barreiro, Álvaro
Format:	Preprint
Published:	2024
Subjects:	Information Retrieval
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Towards Reliable Testing for Multiple Information Retrieval System Comparisons
by: Otero, David, et al.
Published: (2025)

LLM-Assisted Pseudo-Relevance Feedback
by: Otero, David, et al.
Published: (2026)

Hybrid Pooling with LLMs via Relevance Context Learning
by: Otero, David, et al.
Published: (2026)

Enhancing Automatic Keyphrase Labelling with Text-to-Text Transfer Transformer (T5) Architecture: A Framework for Keyphrase Generation and Filtering
by: Gabín, Jorge, et al.
Published: (2024)

JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
by: Rahmani, Hossein A., et al.
Published: (2024)

Beyond Questions: Leveraging ColBERT for Keyphrase Search
by: Gabín, Jorge, et al.
Published: (2024)

Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG
by: Gabín, Jorge, et al.
Published: (2026)

Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance
by: Rampisela, Theresia Veronika, et al.
Published: (2024)

Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models
by: Zhou, Jianqun, et al.
Published: (2024)

Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier
by: Rampisela, Theresia Veronika, et al.
Published: (2025)

Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval
by: Ma, Shengjie, et al.
Published: (2024)

LLM4Tag: Automatic Tagging System for Information Retrieval via Large Language Models
by: Tang, Ruiming, et al.
Published: (2025)

A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look
by: Upadhyay, Shivani, et al.
Published: (2024)

The Cranfield II Relevance Assessments: A Critical Evaluation
by: Harter, Stephen P.
Published: (1971)

CFaiRLLM: Consumer Fairness Evaluation in Large-Language Model Recommender System
by: Deldjoo, Yashar, et al.
Published: (2024)

Exploring Large Language Models for Relevance Judgments in Tetun
by: de Jesus, Gabriel, et al.
Published: (2024)

Enhancing Spectral Knowledge Interrogation: A Reliable Retrieval-Augmented Generative Framework on Large Language Models
by: Liang, Jiheng, et al.
Published: (2024)

Toward Automatic Group Membership Annotation for Group Fairness Evaluation
by: Chen, Fumian, et al.
Published: (2024)

Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation
by: Yang, Jheng-Hong, et al.
Published: (2024)

An Evaluation Framework for Attributed Information Retrieval using Large Language Models
by: Djeddal, Hanane, et al.
Published: (2024)

Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrieval
by: Chun, Yongchan, et al.
Published: (2025)

Understanding the Role of Cross-Entropy Loss in Fairly Evaluating Large Language Model-based Recommendation
by: Xu, Cong, et al.
Published: (2024)

Breaking the Lens of the Telescope: Online Relevance Estimation over Large Retrieval Sets
by: Rathee, Mandeep, et al.
Published: (2025)

Retrieval-in-the-Chain: Bootstrapping Large Language Models for Generative Retrieval
by: Zhang, Yingchen, et al.
Published: (2025)

Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing
by: Yan, Le, et al.
Published: (2024)

Evaluating Large Language Models for Cross-Lingual Retrieval
by: Zuo, Longfei, et al.
Published: (2025)

Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness
by: Zhao, Xinran, et al.
Published: (2024)

Generative Retrieval Meets Multi-Graded Relevance
by: Tang, Yubao, et al.
Published: (2024)

Is Relevance Propagated from Retriever to Generator in RAG?
by: Tian, Fangzheng, et al.
Published: (2025)

Language Fairness in Multilingual Information Retrieval
by: Yang, Eugene, et al.
Published: (2024)

Automatic Classification in Information Retrieval.
by: van Rijsbergen, C. J.
Published: (1978)

Item-side Fairness of Large Language Model-based Recommendation System
by: Jiang, Meng, et al.
Published: (2024)

Axiomatic Causal Interventions for Reverse Engineering Relevance Computation in Neural Retrieval Models
by: Chen, Catherine, et al.
Published: (2024)

Discovering Biases in Information Retrieval Models Using Relevance Thesaurus as Global Explanation
by: Kim, Youngwoo, et al.
Published: (2024)

GenTREC: The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems
by: Türkmen, Mehmet Deniz, et al.
Published: (2025)

Towards More Relevant Product Search Ranking Via Large Language Models: An Empirical Study
by: Liu, Qi, et al.
Published: (2024)

Graded Relevance Scoring of Written Essays with Dense Retrieval
by: Albatarni, Salam, et al.
Published: (2024)

Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarios
by: Caramancion, Kevin Matthe
Published: (2024)

Towards FairRAG: Preventing Representational Harm in Retrieval-Augmented Generation by Enforcing Fair Exposure at Retrieval Time
by: Tikoo, Riddhi
Published: (2026)

Improving Pinterest Search Relevance Using Large Language Models
by: Wang, Han, et al.
Published: (2024)