Image de couverture de livre

When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Yu, Chuting, Li, Hang, Zuccon, Guido, Mackenzie, Joel, Leelanupab, Teerapong
Format:	Preprint
Publié:	2026
Sujets:	Information Retrieval
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Whole-Pool Setwise Reranking with Long-Context Language Models
par: Li, Hang, et autres
Publié: (2026)

DenseReviewer: A Screening Prioritisation Tool for Systematic Review based on Dense Retrieval
par: Mao, Xinyu, et autres
Publié: (2025)

Beyond GeneGPT: A Multi-Agent Architecture with Open-Source LLMs for Enhanced Genomic Question Answering
par: Chen, Haodong, et autres
Publié: (2025)

Where Relevance Emerges: A Layer-Wise Study of Internal Attention for Zero-Shot Re-Ranking
par: Chen, Haodong, et autres
Publié: (2026)

AiReview: An Open Platform for Accelerating Systematic Reviews with LLMs
par: Mao, Xinyu, et autres
Publié: (2025)

Embark on DenseQuest: A System for Selecting the Best Dense Retriever for a Custom Collection
par: Khramtsova, Ekaterina, et autres
Publié: (2024)

TPRF: A Transformer-based Pseudo-Relevance Feedback Model for Efficient and Effective Retrieval
par: Li, Hang, et autres
Publié: (2024)

LLM-VPRF: Large Language Model Based Vector Pseudo Relevance Feedback
par: Li, Hang, et autres
Publié: (2025)

Pseudo Relevance Feedback is Enough to Close the Gap Between Small and Large Dense Retrieval Models
par: Li, Hang, et autres
Publié: (2025)

Judging the Judges: A Collection of LLM-Generated Relevance Judgements
par: Rahmani, Hossein A., et autres
Publié: (2025)

AutoBool: An Reinforcement-Learning trained LLM for Effective Automated Boolean Query Generation for Systematic Reviews
par: Wang, Shuai, et autres
Publié: (2025)

JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
par: Rahmani, Hossein A., et autres
Publié: (2024)

Team IELAB at TREC Clinical Trial Track 2023: Enhancing Clinical Trial Retrieval with Neural Rankers and Large Language Models
par: Zhuang, Shengyao, et autres
Publié: (2024)

Does Vec2Text Pose a New Corpus Poisoning Threat?
par: Zhuang, Shengyao, et autres
Publié: (2024)

A Reproducibility Study of Goldilocks: Just-Right Tuning of BERT for TAR
par: Mao, Xinyu, et autres
Publié: (2024)

Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment
par: Chen, Nuo, et autres
Publié: (2026)

An Investigation of Prompt Variations for Zero-shot LLM-based Rankers
par: Sun, Shuoqi, et autres
Publié: (2024)

Beyond Chunk-Then-Embed: A Comprehensive Taxonomy and Evaluation of Document Chunking Strategies for Information Retrieval
par: Zhou, Yongjie, et autres
Publié: (2026)

Leveraging LLMs for Unsupervised Dense Retriever Ranking
par: Khramtsova, Ekaterina, et autres
Publié: (2024)

Dense Retrieval with Continuous Explicit Feedback for Systematic Review Screening Prioritisation
par: Mao, Xinyu, et autres
Publié: (2024)

Large Language Models for Stemming: Promises, Pitfalls and Failures
par: Wang, Shuai, et autres
Publié: (2024)

Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition
par: Yao, Zheng, et autres
Publié: (2025)

Illusions of Relevance: Arbitrary Content Injection Attacks Deceive Retrievers, Rerankers, and LLM Judges
par: Tamber, Manveer Singh, et autres
Publié: (2025)

Can It Reach the Generator? Investigating the Survival of Prompt-Injection Attacks in Realistic RAG Settings
par: Yin, Yu, et autres
Publié: (2026)

To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay
par: Dey, Soumik, et autres
Publié: (2025)

Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning
par: Zhuang, Shengyao, et autres
Publié: (2025)

Is Architectural Complexity Overrated? Competitive and Interpretable Knowledge Graph Completion with RelatE
par: Chakraborty, Abhijit, et autres
Publié: (2025)

How Much Freedom Does An Effectiveness Metric Really Have?
par: Moffat, Alistair, et autres
Publié: (2023)

PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
par: Zhuang, Shengyao, et autres
Publié: (2024)

Starbucks-v2: Improved Training for 2D Matryoshka Embeddings
par: Zhuang, Shengyao, et autres
Publié: (2024)

Batched Self-Consistency Improves LLM Relevance Assessment and Ranking
par: Korikov, Anton, et autres
Publié: (2025)

Revisiting Human-vs-LLM judgments using the TREC Podcast Track
par: Mansour, Watheq, et autres
Publié: (2026)

Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents
par: Wang, Haoyu, et autres
Publié: (2025)

REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking
par: Wang, Pinhuan, et autres
Publié: (2025)

Inferential Question Answering
par: Mozafari, Jamshid, et autres
Publié: (2026)

Unlearning for Federated Online Learning to Rank: A Reproducibility Study
par: Tao, Yiling, et autres
Publié: (2025)

ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search
par: Wang, Shuai, et autres
Publié: (2024)

2D Matryoshka Training for Information Retrieval
par: Wang, Shuai, et autres
Publié: (2024)

Understanding and Mitigating the Threat of Vec2Text to Dense Retrieval Systems
par: Zhuang, Shengyao, et autres
Publié: (2024)

Reassessing Large Language Model Boolean Query Generation for Systematic Reviews
par: Wang, Shuai, et autres
Publié: (2025)