Copertina

When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Yu, Chuting, Li, Hang, Zuccon, Guido, Mackenzie, Joel, Leelanupab, Teerapong
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Information Retrieval
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Whole-Pool Setwise Reranking with Long-Context Language Models
di: Li, Hang, et al.
Pubblicazione: (2026)

DenseReviewer: A Screening Prioritisation Tool for Systematic Review based on Dense Retrieval
di: Mao, Xinyu, et al.
Pubblicazione: (2025)

Beyond GeneGPT: A Multi-Agent Architecture with Open-Source LLMs for Enhanced Genomic Question Answering
di: Chen, Haodong, et al.
Pubblicazione: (2025)

Where Relevance Emerges: A Layer-Wise Study of Internal Attention for Zero-Shot Re-Ranking
di: Chen, Haodong, et al.
Pubblicazione: (2026)

AiReview: An Open Platform for Accelerating Systematic Reviews with LLMs
di: Mao, Xinyu, et al.
Pubblicazione: (2025)

Embark on DenseQuest: A System for Selecting the Best Dense Retriever for a Custom Collection
di: Khramtsova, Ekaterina, et al.
Pubblicazione: (2024)

TPRF: A Transformer-based Pseudo-Relevance Feedback Model for Efficient and Effective Retrieval
di: Li, Hang, et al.
Pubblicazione: (2024)

LLM-VPRF: Large Language Model Based Vector Pseudo Relevance Feedback
di: Li, Hang, et al.
Pubblicazione: (2025)

Pseudo Relevance Feedback is Enough to Close the Gap Between Small and Large Dense Retrieval Models
di: Li, Hang, et al.
Pubblicazione: (2025)

Judging the Judges: A Collection of LLM-Generated Relevance Judgements
di: Rahmani, Hossein A., et al.
Pubblicazione: (2025)

AutoBool: An Reinforcement-Learning trained LLM for Effective Automated Boolean Query Generation for Systematic Reviews
di: Wang, Shuai, et al.
Pubblicazione: (2025)

JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
di: Rahmani, Hossein A., et al.
Pubblicazione: (2024)

Team IELAB at TREC Clinical Trial Track 2023: Enhancing Clinical Trial Retrieval with Neural Rankers and Large Language Models
di: Zhuang, Shengyao, et al.
Pubblicazione: (2024)

Does Vec2Text Pose a New Corpus Poisoning Threat?
di: Zhuang, Shengyao, et al.
Pubblicazione: (2024)

A Reproducibility Study of Goldilocks: Just-Right Tuning of BERT for TAR
di: Mao, Xinyu, et al.
Pubblicazione: (2024)

Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment
di: Chen, Nuo, et al.
Pubblicazione: (2026)

An Investigation of Prompt Variations for Zero-shot LLM-based Rankers
di: Sun, Shuoqi, et al.
Pubblicazione: (2024)

Beyond Chunk-Then-Embed: A Comprehensive Taxonomy and Evaluation of Document Chunking Strategies for Information Retrieval
di: Zhou, Yongjie, et al.
Pubblicazione: (2026)

Leveraging LLMs for Unsupervised Dense Retriever Ranking
di: Khramtsova, Ekaterina, et al.
Pubblicazione: (2024)

Dense Retrieval with Continuous Explicit Feedback for Systematic Review Screening Prioritisation
di: Mao, Xinyu, et al.
Pubblicazione: (2024)

Large Language Models for Stemming: Promises, Pitfalls and Failures
di: Wang, Shuai, et al.
Pubblicazione: (2024)

Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition
di: Yao, Zheng, et al.
Pubblicazione: (2025)

Illusions of Relevance: Arbitrary Content Injection Attacks Deceive Retrievers, Rerankers, and LLM Judges
di: Tamber, Manveer Singh, et al.
Pubblicazione: (2025)

Can It Reach the Generator? Investigating the Survival of Prompt-Injection Attacks in Realistic RAG Settings
di: Yin, Yu, et al.
Pubblicazione: (2026)

To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay
di: Dey, Soumik, et al.
Pubblicazione: (2025)

Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning
di: Zhuang, Shengyao, et al.
Pubblicazione: (2025)

Is Architectural Complexity Overrated? Competitive and Interpretable Knowledge Graph Completion with RelatE
di: Chakraborty, Abhijit, et al.
Pubblicazione: (2025)

How Much Freedom Does An Effectiveness Metric Really Have?
di: Moffat, Alistair, et al.
Pubblicazione: (2023)

PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
di: Zhuang, Shengyao, et al.
Pubblicazione: (2024)

Starbucks-v2: Improved Training for 2D Matryoshka Embeddings
di: Zhuang, Shengyao, et al.
Pubblicazione: (2024)

Batched Self-Consistency Improves LLM Relevance Assessment and Ranking
di: Korikov, Anton, et al.
Pubblicazione: (2025)

Revisiting Human-vs-LLM judgments using the TREC Podcast Track
di: Mansour, Watheq, et al.
Pubblicazione: (2026)

Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents
di: Wang, Haoyu, et al.
Pubblicazione: (2025)

REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking
di: Wang, Pinhuan, et al.
Pubblicazione: (2025)

Inferential Question Answering
di: Mozafari, Jamshid, et al.
Pubblicazione: (2026)

Unlearning for Federated Online Learning to Rank: A Reproducibility Study
di: Tao, Yiling, et al.
Pubblicazione: (2025)

ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search
di: Wang, Shuai, et al.
Pubblicazione: (2024)

2D Matryoshka Training for Information Retrieval
di: Wang, Shuai, et al.
Pubblicazione: (2024)

Understanding and Mitigating the Threat of Vec2Text to Dense Retrieval Systems
di: Zhuang, Shengyao, et al.
Pubblicazione: (2024)

Reassessing Large Language Model Boolean Query Generation for Systematic Reviews
di: Wang, Shuai, et al.
Pubblicazione: (2025)