Cover Image

Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Tian, Zailong, Han, Zhuoheng, Chen, Yanzhe, Xu, Haozhe, Yang, Xi, Xuan, Richeng, Wang, Houfeng, Liao, Lizi
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting
by: Tian, Zailong, et al.
Published: (2026)

Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement
by: Lu, Junyu, et al.
Published: (2025)

Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning
by: Xu, Yuanda, et al.
Published: (2026)

Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMs
by: Xu, Chenjun, et al.
Published: (2025)

Aligned Multi-View Scripts for Universal Chart-to-Code Generation
by: Zhang, Zhihan, et al.
Published: (2026)

Investigating More Explainable and Partition-Free Compositionality Estimation for LLMs: A Rule-Generation Perspective
by: Xu, Ziyao, et al.
Published: (2026)

Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models
by: Chhikara, Prateek
Published: (2025)

LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval
by: Epstein, Elliot L., et al.
Published: (2025)

Not Wrong, But Untrue: LLM Overconfidence in Document-Based Queries
by: Hagar, Nick, et al.
Published: (2025)

Boosting Chart-to-Code Generation in MLLM via Dual Preference-Guided Refinement
by: Zhang, Zhihan, et al.
Published: (2025)

Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement
by: Zhou, Xiaofeng, et al.
Published: (2025)

Retrieval Augmented Generation for Dynamic Graph Modeling
by: Wu, Yuxia, et al.
Published: (2024)

Searching for Privacy Risks in LLM Agents via Simulation
by: Zhang, Yanzhe, et al.
Published: (2025)

RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following
by: Pan, Tianjun, et al.
Published: (2026)

Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning
by: Tang, Lexiang, et al.
Published: (2026)

JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
by: Shi, Zhichao, et al.
Published: (2025)

Extended to Reality: Prompt Injection in 3D Environments
by: Li, Zhuoheng, et al.
Published: (2026)

LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspection
by: Hossain, Akram, et al.
Published: (2026)

Agentic Uncertainty Reveals Agentic Overconfidence
by: Kaddour, Jean, et al.
Published: (2026)

PCQPR: Proactive Conversational Question Planning with Reflection
by: Guo, Shasha, et al.
Published: (2024)

BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge
by: Tong, Terry, et al.
Published: (2025)

Trust the uncertain teacher: distilling dark knowledge via calibrated uncertainty
by: Kim, Jeonghyun, et al.
Published: (2026)

Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge
by: Pan, Bo, et al.
Published: (2026)

SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation
by: Guo, Shasha, et al.
Published: (2024)

Static IR Drop Prediction with Attention U-Net and Saliency-Based Explainability
by: Zhang, Lizi, et al.
Published: (2024)

A Survey on Neural Question Generation: Methods, Applications, and Prospects
by: Guo, Shasha, et al.
Published: (2024)

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
by: Ye, Jiayi, et al.
Published: (2024)

Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge
by: Chen, Luyu, et al.
Published: (2025)

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
by: Wang, Yidong, et al.
Published: (2025)

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
by: Yuan, Tongxin, et al.
Published: (2024)

Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systems
by: Li, Xiaochuan, et al.
Published: (2025)

Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
by: Han, Steve, et al.
Published: (2025)

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
by: Dev, Sunishchal, et al.
Published: (2026)

Multi-Agent Debate for LLM Judges with Adaptive Stability Detection
by: Hu, Tianyu, et al.
Published: (2025)

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
by: He, Zheqi, et al.
Published: (2024)

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
by: Soumik, Sadman Kabir
Published: (2026)

UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-Judge
by: Zhang, Yang, et al.
Published: (2025)

Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosis
by: Wang, Haochun, et al.
Published: (2024)

MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation
by: Wang, Yutong, et al.
Published: (2025)

CatNet: Controlling the False Discovery Rate in LSTM with SHAP Feature Importance and Gaussian Mirrors
by: Han, Jiaan, et al.
Published: (2024)