Copertina

Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Schroeder, Kayla, Wood-Doughty, Zach
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Reliability of Topic Modeling
di: Schroeder, Kayla, et al.
Pubblicazione: (2024)

Improving LLM-as-a-Judge Inference with the Judgment Distribution
di: Wang, Victor, et al.
Pubblicazione: (2025)

Can You Trick the Grader? Adversarial Persuasion of LLM Judges
di: Hwang, Yerin, et al.
Pubblicazione: (2025)

Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
di: Wang, Leyao, et al.
Pubblicazione: (2026)

How Reliable is Multilingual LLM-as-a-Judge?
di: Fu, Xiyan, et al.
Pubblicazione: (2025)

Can LLM be a Personalized Judge?
di: Dong, Yijiang River, et al.
Pubblicazione: (2024)

Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge
di: Chen, Luyu, et al.
Pubblicazione: (2025)

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
di: Wang, Yidong, et al.
Pubblicazione: (2025)

Can We Trust LLM Detectors?
di: Sandhan, Jivnesh, et al.
Pubblicazione: (2026)

VERT: Reliable LLM Judges for Radiology Report Evaluation
di: Bologna, Federica, et al.
Pubblicazione: (2026)

Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
di: Jung, Jaehun, et al.
Pubblicazione: (2024)

LLM-REVal: Can We Trust LLM Reviewers Yet?
di: Li, Rui, et al.
Pubblicazione: (2025)

AI vs. Human Judgment of Content Moderation: LLM-as-a-Judge and Ethics-Based Response Refusals
di: Pasch, Stefan
Pubblicazione: (2025)

LLM-as-a-Judge & Reward Model: What They Can and Cannot Do
di: Son, Guijin, et al.
Pubblicazione: (2024)

SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge?
di: Chen, Jiamin, et al.
Pubblicazione: (2026)

When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation
di: Badawi, Abeer, et al.
Pubblicazione: (2025)

An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability
di: Yamauchi, Yusuke, et al.
Pubblicazione: (2025)

Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
di: Sun, Xin, et al.
Pubblicazione: (2026)

Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation
di: Lin, Wei-Hsiang, et al.
Pubblicazione: (2025)

The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
di: Marioriyad, Arash, et al.
Pubblicazione: (2025)

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
di: Yang, Bo, et al.
Pubblicazione: (2026)

Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
di: Findeis, Arduin, et al.
Pubblicazione: (2025)

Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
di: Zhang, Xingjian, et al.
Pubblicazione: (2025)

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
di: Gupta, Manan, et al.
Pubblicazione: (2026)

Beyond the Surface: Measuring Self-Preference in LLM Judgments
di: Chen, Zhi-Yuan, et al.
Pubblicazione: (2025)

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment
di: Ferrer, Robinson, et al.
Pubblicazione: (2026)

Quantitative LLM Judges
di: Sahoo, Aishwarya, et al.
Pubblicazione: (2025)

Can You Make It Sound Like You? Post-Editing LLM-Generated Text for Personal Style
di: Baumler, Connor, et al.
Pubblicazione: (2026)

A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
di: Schwinn, Leo, et al.
Pubblicazione: (2026)

Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA
di: Belmadani, Ikram, et al.
Pubblicazione: (2026)

EnsemJudge: Enhancing Reliability in Chinese LLM-Generated Text Detection through Diverse Model Ensembles
di: Wang, Zhuoshang, et al.
Pubblicazione: (2026)

Self-Preference Bias in LLM-as-a-Judge
di: Wataoka, Koki, et al.
Pubblicazione: (2024)

The Necessity of Setting Temperature in LLM-as-a-Judge
di: Li, Lujun, et al.
Pubblicazione: (2026)

Evaluating Scoring Bias in LLM-as-a-Judge
di: Li, Qingquan, et al.
Pubblicazione: (2025)

Benchmarking LLM-based Relevance Judgment Methods
di: Arabzadeh, Negar, et al.
Pubblicazione: (2025)

An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4
di: Huang, Hui, et al.
Pubblicazione: (2024)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
di: Shi, Lin, et al.
Pubblicazione: (2024)

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
di: Bellibatlu, Rohith Reddy, et al.
Pubblicazione: (2026)

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
di: Hong, Yihan, et al.
Pubblicazione: (2026)

A Survey on LLM-as-a-Judge
di: Gu, Jiawei, et al.
Pubblicazione: (2024)