Cover Image

Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluations

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Jain, Suryaansh, Ahmed, Umair Z., Sahai, Shubham, Leong, Ben
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
by: Soumik, Sadman Kabir
Published: (2026)

Measuring Agreeableness Bias in Multimodal Models
by: Lim, Jaehyuk, et al.
Published: (2024)

Quantifying and Mitigating Self-Preference Bias of LLM Judges
by: Yang, Jinming, et al.
Published: (2026)

Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge
by: Fujinuma, Yoshinari
Published: (2025)

Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systems
by: Li, Xiaochuan, et al.
Published: (2025)

Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation
by: Feuer, Benjamin, et al.
Published: (2026)

Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
by: Zhang, Hongbin, et al.
Published: (2026)

BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation
by: Lai, Peng, et al.
Published: (2026)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
by: Shi, Lin, et al.
Published: (2024)

Permutation-Consensus Listwise Judging for Robust Factuality Evaluation
by: Huang, Tianyi, et al.
Published: (2026)

VERT: Reliable LLM Judges for Radiology Report Evaluation
by: Bologna, Federica, et al.
Published: (2026)

Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems
by: Gao, Jiaxin, et al.
Published: (2025)

Beyond Self-learned Attention: Mitigating Attention Bias in Transformer-based Models Using Attention Guidance
by: Gesi, Jiri, et al.
Published: (2024)

Causal Intervention-Based Memory Selection for Long-Horizon LLM Agents
by: Srivastava, Saksham Sahai
Published: (2026)

Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning
by: Wu, Xuyang, et al.
Published: (2025)

JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
by: Shi, Zhichao, et al.
Published: (2025)

Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
by: Zhao, Zixiao, et al.
Published: (2026)

Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity
by: Yosef, Erez, et al.
Published: (2026)

REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
by: Zhang, Yasi, et al.
Published: (2026)

Subgroups Matter for Robust Bias Mitigation
by: Alloula, Anissa, et al.
Published: (2025)

Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge
by: Zhou, Xiaolin, et al.
Published: (2026)

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation
by: Alam, Firoj, et al.
Published: (2026)

Evaluating Metrics for Safety with LLM-as-Judges
by: Clegg, Kester, et al.
Published: (2025)

Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judge
by: Ma, Chiyu, et al.
Published: (2025)

Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment
by: Tan, Renxuan, et al.
Published: (2026)

Bias Testing and Mitigation in LLM-based Code Generation
by: Huang, Dong, et al.
Published: (2023)

Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge
by: Elangovan, Aparna, et al.
Published: (2024)

Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfalls
by: Fandina, Ora Nova, et al.
Published: (2025)

Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Models
by: Shah, Arya, et al.
Published: (2026)

JudgeBench: A Benchmark for Evaluating LLM-based Judges
by: Tan, Sijun, et al.
Published: (2024)

UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulation
by: Zhou, Hanzhang, et al.
Published: (2024)

On Evaluating LLM Alignment by Evaluating LLMs as Judges
by: Liu, Yixin, et al.
Published: (2025)

AI-Driven Agents with Prompts Designed for High Agreeableness Increase the Likelihood of Being Mistaken for a Human in the Turing Test
by: León-Domínguez, U., et al.
Published: (2024)

Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
by: Spiliopoulou, Evangelia, et al.
Published: (2025)

Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models
by: Kumar, Shachi H, et al.
Published: (2024)

Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge
by: Chen, Luyu, et al.
Published: (2025)

MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation
by: Wang, Yutong, et al.
Published: (2025)

Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking
by: Vardasbi, Ali, et al.
Published: (2025)

AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory
by: Jain, Jitesh, et al.
Published: (2025)

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
by: Lee, Sua, et al.
Published: (2026)