Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Haldar, Rajarshi, Hockenmaier, Julia
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Analyzing the Performance of Large Language Models on Code Summarization
by: Haldar, Rajarshi, et al.
Published: (2024)

A Multi-Perspective Architecture for Semantic Code Search
by: Haldar, Rajarshi, et al.
Published: (2020)

Evaluating Step-by-step Reasoning Traces: A Survey
by: Lee, Jinu, et al.
Published: (2025)

Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures
by: Sidhu, Risham, et al.
Published: (2026)

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
by: Wang, Yidong, et al.
Published: (2025)

LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories
by: Vishnubhotla, Krishnapriya, et al.
Published: (2026)

Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge
by: Lau, Fiona
Published: (2026)

Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing
by: Ahn, Jihyun Janice, et al.
Published: (2025)

Dynamically Allocated Interval-Based Generative Linguistic Steganography with Roulette Wheel
by: Wang, Yihao, et al.
Published: (2024)

ReasoningFlow: Semantic Structure of Complex Reasoning Traces
by: Lee, Jinu, et al.
Published: (2025)

Self-Preference Bias in LLM-as-a-Judge
by: Wataoka, Koki, et al.
Published: (2024)

How Reliable are Causal Probing Interventions?
by: Canby, Marc, et al.
Published: (2024)

Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics
by: Lee, Jinu, et al.
Published: (2025)

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
by: Tapwal, Riya, et al.
Published: (2026)

SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
by: Yegorova, Yekaterina, et al.
Published: (2026)

Misleading through Inconsistency: A Benchmark for Political Inconsistencies Detection
by: Sagimbayeva, Nursulu, et al.
Published: (2025)

The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
by: Marioriyad, Arash, et al.
Published: (2025)

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
by: Yang, Bo, et al.
Published: (2026)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
by: Shi, Lin, et al.
Published: (2024)

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
by: Bellibatlu, Rohith Reddy, et al.
Published: (2026)

Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA
by: Belmadani, Ikram, et al.
Published: (2026)

RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learning
by: Huang, Jerry, et al.
Published: (2025)

Entailment-Preserving First-order Logic Representations in Natural Language Entailment
by: Lee, Jinu, et al.
Published: (2025)

How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflows
by: Han, Songhee, et al.
Published: (2026)

Quantifying and Mitigating Self-Preference Bias of LLM Judges
by: Yang, Jinming, et al.
Published: (2026)

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
by: Zhu, Ziyi, et al.
Published: (2026)

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
by: Tang, Zhenwei, et al.
Published: (2026)

A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
by: Li, Chenyu, et al.
Published: (2026)

Quantitative LLM Judges
by: Sahoo, Aishwarya, et al.
Published: (2025)

BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
by: Jayannavar, Prashant, et al.
Published: (2025)

Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
by: Spiliopoulou, Evangelia, et al.
Published: (2025)

Can LLM be a Personalized Judge?
by: Dong, Yijiang River, et al.
Published: (2024)

SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification
by: Yoon, Kanghoon, et al.
Published: (2025)

A Survey on LLM-as-a-Judge
by: Gu, Jiawei, et al.
Published: (2024)

Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias
by: Tonneau, Manuel, et al.
Published: (2026)

Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluation
by: Lee, Dongryeol, et al.
Published: (2026)

SIFiD: Reassess Summary Factual Inconsistency Detection with LLM
by: Yang, Jiuding, et al.
Published: (2024)

On Finding Inconsistencies in Documents
by: Lovering, Charles J., et al.
Published: (2025)

Self-Judge: Selective Instruction Following with Alignment Self-Evaluation
by: Ye, Hai, et al.
Published: (2024)

Evaluating Scoring Bias in LLM-as-a-Judge
by: Li, Qingquan, et al.
Published: (2025)