Image de couverture de livre

Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Lau, Fiona
Format:	Preprint
Publié:	2026
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
par: Kostić, Bogdan, et autres
Publié: (2026)

Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
par: Haldar, Rajarshi, et autres
Publié: (2025)

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
par: Wang, Yidong, et autres
Publié: (2025)

LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories
par: Vishnubhotla, Krishnapriya, et autres
Publié: (2026)

Evaluating Scoring Bias in LLM-as-a-Judge
par: Li, Qingquan, et autres
Publié: (2025)

Same Content, Different Representations: A Controlled Study for Table QA
par: Zhang, Yue, et autres
Publié: (2025)

Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias
par: Tonneau, Manuel, et autres
Publié: (2026)

Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering
par: Molfese, Francesco Maria, et autres
Publié: (2025)

Personalized LLM for Generating Customized Responses to the Same Query from Different Users
par: Zeng, Hang, et autres
Publié: (2024)

The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling
par: Zhang, Ruochen, et autres
Publié: (2024)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
par: Shi, Lin, et autres
Publié: (2024)

AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge
par: Zhou, Karen, et autres
Publié: (2026)

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
par: Tang, Zhenwei, et autres
Publié: (2026)

An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4
par: Huang, Hui, et autres
Publié: (2024)

Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge
par: Fujinuma, Yoshinari
Publié: (2025)

Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing
par: Ahn, Jihyun Janice, et autres
Publié: (2025)

Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models
par: Lin, Zizhuo, et autres
Publié: (2026)

Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transport
par: Morandi, Andrea
Publié: (2026)

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
par: Levy, Mosh, et autres
Publié: (2024)

Summarization Metrics for Spanish and Basque: Do Automatic Scores and LLM-Judges Correlate with Humans?
par: Barnes, Jeremy, et autres
Publié: (2025)

Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction
par: Kamoi, Ryo, et autres
Publié: (2026)

Misleading through Inconsistency: A Benchmark for Political Inconsistencies Detection
par: Sagimbayeva, Nursulu, et autres
Publié: (2025)

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
par: Hong, Yihan, et autres
Publié: (2026)

When LLM Judge Scores Look Good but Best-of-N Decisions Fail
par: Landesberg, Eddie
Publié: (2026)

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
par: Yang, Bo, et autres
Publié: (2026)

The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
par: Marioriyad, Arash, et autres
Publié: (2025)

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
par: Bellibatlu, Rohith Reddy, et autres
Publié: (2026)

Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
par: Wu, Junjie, et autres
Publié: (2026)

EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models
par: Su, Jiamin, et autres
Publié: (2025)

Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG
par: Li, Yubo, et autres
Publié: (2026)

Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA
par: Belmadani, Ikram, et autres
Publié: (2026)

Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs
par: Ford, Casey, et autres
Publié: (2026)

AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling
par: Miao, Yongliang, et autres
Publié: (2026)

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
par: Zhu, Ziyi, et autres
Publié: (2026)

AmbigDocs: Reasoning across Documents on Different Entities under the Same Name
par: Lee, Yoonsang, et autres
Publié: (2024)

Input Order Shapes LLM Semantic Alignment in Multi-Document Summarization
par: Ma, Jing
Publié: (2025)

Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models
par: Le, Hieu Xuan, et autres
Publié: (2026)

Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs
par: Alkaeed, Mahdi, et autres
Publié: (2026)

Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges
par: Tang, Yuqi, et autres
Publié: (2025)

Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM
par: Liang, Jingcong, et autres
Publié: (2024)