Cover Image

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Rao, Delip, Callison-Burch, Chris
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Human-Computer Interaction Data Analysis, Statistics and Probability
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Toward Ethical Spatial Analysis: Addressing Endogenous Bias Through Visual Analytics
by: Chen, Chuan, et al.
Published: (2024)

Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap
by: Zhu, Andrew, et al.
Published: (2025)

Human-aligned Quantification of Numerical Data
by: Kolonin, Anton
Published: (2025)

Autorubric: Unifying Rubric-based LLM Evaluation
by: Rao, Delip, et al.
Published: (2026)

First Steps Towards Overhearing LLM Agents: A Case Study With Dungeons & Dragons Gameplay
by: Zhu, Andrew, et al.
Published: (2025)

What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis
by: Rao, Delip, et al.
Published: (2026)

BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation
by: Rao, Delip, et al.
Published: (2026)

CALYPSO: LLMs as Dungeon Masters' Assistants
by: Zhu, Andrew, et al.
Published: (2023)

Toward Beginner-Friendly LLMs for Language Learning: Controlling Difficulty in Conversation
by: Jin, Meiqing, et al.
Published: (2025)

Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT
by: Javanmard, Younes, et al.
Published: (2026)

Scale-Dependent Semantic Dynamics Revealed by Allan Deviation
by: Dasgupta, Debayan
Published: (2026)

Extracting ORR Catalyst Information for Fuel Cell from Scientific Literature
by: Htet, Hein, et al.
Published: (2025)

Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
by: Rao, Delip, et al.
Published: (2026)

ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models
by: Rao, Delip, et al.
Published: (2026)

Automated Review Generation Method Based on Large Language Models
by: Wu, Shican, et al.
Published: (2024)

Identification of the parameters of complex constitutive models: Least squares minimization vs. Bayesian updating
by: Most, Thomas
Published: (2024)

Exploring language relations through syntactic distances and geographic proximity
by: De Gregorio, Juan, et al.
Published: (2024)

Approaches to the classification of complex systems: Words, texts, and more
by: Rovenchak, Andrij
Published: (2022)

Precision Evaluation Criteria for Simulation Algorithms in Infinite Systems: A Network Model-Based Approach
by: Ding, Yonglong
Published: (2024)

GPU-Accelerated Sequential Monte Carlo for Bayesian Spectral Analysis
by: Nabika, Tomohiro, et al.
Published: (2026)

`pandemonium`: High Dimensional Analysis in Linked Spaces
by: McCoy, Gabriel, et al.
Published: (2026)

PolyStan: PolyChord nested sampling and Bayesian evidences for Stan models
by: Fowlie, Andrew
Published: (2025)

On computation of a common mean
by: Malkin, Zinovy
Published: (2011)

Novel computational approaches for ratio distributions with an application to Hake's ratio in effect size measurement
by: Hanč, Jozef, et al.
Published: (2024)

Inverse Uncertainty Quantification by Hierarchical Bayesian Modeling and Application in Nuclear System Thermal-Hydraulics Codes
by: Wang, Chen, et al.
Published: (2023)

NSF-SciFy: Mining the NSF Awards Database for Scientific Claims
by: Rao, Delip, et al.
Published: (2025)

Sampling parameters of ordinary differential equations with Langevin dynamics that satisfy constraints
by: Chi, Chris, et al.
Published: (2024)

AI vs. Human Judgment of Content Moderation: LLM-as-a-Judge and Ethics-Based Response Refusals
by: Pasch, Stefan
Published: (2025)

ELI-Why: Evaluating the Pedagogical Utility of Language Model Explanations
by: Joshi, Brihi, et al.
Published: (2025)

PRISM: Deriving a White-Box Transformer as a Signal-Noise Decomposition Operator via Maximum Coding Rate Reduction
by: Huang, Dongchen
Published: (2026)

Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play
by: Ye, Ziyu, et al.
Published: (2024)

Toward a statistical mechanics of four letter words
by: Stephens, Greg J., et al.
Published: (2007)

QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums
by: Rao, Varun Nagaraj, et al.
Published: (2024)

Liouville Flow Importance Sampler
by: Tian, Yifeng, et al.
Published: (2024)

When Verification Fails: How Compositionally Infeasible Claims Escape Rejection
by: Liu, Muxin, et al.
Published: (2026)

PoissonRatioUQ: An R package for band ratio uncertainty quantification
by: LeDuc, Matthew, et al.
Published: (2026)

Bayesian Learning in Structural Dynamics: A Comprehensive Review and Emerging Trends
by: Yan, Wang-Ji, et al.
Published: (2025)

Dimensional homogeneity constrained gene expression programming for discovering governing equations
by: Ma, Wenjun, et al.
Published: (2022)

Accurate Memory Kernel Extraction from Discretized Time Series Data
by: Tepper, Lucas, et al.
Published: (2024)

Numerical Investigation of Preferential Flow Paths in Enzymatically Induced Calcite Precipitation supported by Bayesian Model Analysis
by: Kohlhaas, Rebecca, et al.
Published: (2025)