Cover Image

Are We on the Right Way to Assessing LLM-as-a-Judge?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Feng, Yuanning, Wang, Sinan, Cheng, Zhengxiang, Wan, Yao, Chen, Dongping
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
by: Chen, Dongping, et al.
Published: (2024)

Optimizing Length Compression in Large Reasoning Models
by: Cheng, Zhengxiang, et al.
Published: (2025)

Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
by: Shen, Wenxuan, et al.
Published: (2025)

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
by: Wang, Chenlong, et al.
Published: (2025)

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
by: Ye, Jiayi, et al.
Published: (2024)

CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom
by: Li, Yisen, et al.
Published: (2025)

Measuring Iterative Temporal Reasoning with Time Puzzles
by: Wang, Zhengxiang, et al.
Published: (2026)

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
by: Wang, Yidong, et al.
Published: (2025)

HonestLLM: Toward an Honest and Helpful Large Language Model
by: Gao, Chujie, et al.
Published: (2024)

BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge
by: Tong, Terry, et al.
Published: (2025)

Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce
by: Chen, Liang, et al.
Published: (2026)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
by: Shi, Lin, et al.
Published: (2024)

Evaluating LLMs with Multiple Problems at once
by: Wang, Zhengxiang, et al.
Published: (2024)

A Survey on LLM-as-a-Judge
by: Gu, Jiawei, et al.
Published: (2024)

Self-Cognition in Large Language Models: An Exploratory Study
by: Chen, Dongping, et al.
Published: (2024)

Can We Trust LLM Detectors?
by: Sandhan, Jivnesh, et al.
Published: (2026)

Wikipedia in the Era of LLMs: Evolution and Risks
by: Huang, Siming, et al.
Published: (2025)

Epistemic Context Learning: Building Trust the Right Way in LLM-Based Multi-Agent Systems
by: Zhou, Ruiwen, et al.
Published: (2026)

Expanding the Scope: Inductive Knowledge Graph Reasoning with Multi-Starting Progressive Propagation
by: Shao, Zhoutian, et al.
Published: (2024)

A Prompt-Based Knowledge Graph Foundation Model for Universal In-Context Reasoning
by: Cui, Yuanning, et al.
Published: (2024)

Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge
by: Chen, Luyu, et al.
Published: (2025)

LLM-as-a-Judge for Time Series Explanations
by: Sivalingam, Preetham, et al.
Published: (2026)

Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
by: Saha, Swarnadeep, et al.
Published: (2025)

JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
by: Shi, Zhichao, et al.
Published: (2025)

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
by: Jiang, Hongchao, et al.
Published: (2025)

JudgeLRM: Large Reasoning Models as a Judge
by: Chen, Nuo, et al.
Published: (2025)

Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic
by: Collot, Stephane, et al.
Published: (2025)

Think-J: Learning to Think for Generative LLM-as-a-Judge
by: Huang, Hui, et al.
Published: (2025)

Evaluating Metrics for Safety with LLM-as-Judges
by: Clegg, Kester, et al.
Published: (2025)

Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
by: Li, Zhuochun, et al.
Published: (2026)

Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiency
by: Wang, Chenlong, et al.
Published: (2025)

Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
by: Zhang, Hongbin, et al.
Published: (2026)

Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge
by: Zhou, Xiaolin, et al.
Published: (2026)

ChatSOS: Vector Database Augmented Generative Question Answering Assistant in Safety Engineering
by: Tang, Haiyang, et al.
Published: (2024)

Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
by: Han, Steve, et al.
Published: (2025)

LLM-REVal: Can We Trust LLM Reviewers Yet?
by: Li, Rui, et al.
Published: (2025)

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
by: Yuan, Tongxin, et al.
Published: (2024)

HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
by: Yang, Langqi, et al.
Published: (2025)

JudgeBench: A Benchmark for Evaluating LLM-based Judges
by: Tan, Sijun, et al.
Published: (2024)

Automated Concept Discovery for LLM-as-a-Judge Preference Analysis
by: Wedgwood, James, et al.
Published: (2026)