How to Correctly Report LLM-as-a-Judge Evaluations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916069689524224 |
|---|---|
| author | Lee, Chungpa Zeng, Thomas Jeong, Jongwon Sohn, Jy-yong Lee, Kangwook |
| author_facet | Lee, Chungpa Zeng, Thomas Jeong, Jongwon Sohn, Jy-yong Lee, Kangwook |
| contents | Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators. However, imperfect sensitivity and specificity of the LLM judges induce bias in naive evaluation scores. We propose a simple plug-in framework that corrects this bias and enables statistically principled uncertainty quantification. Our framework constructs confidence intervals that account for uncertainty from both the test dataset and a human-labeled calibration dataset. Additionally, it uses an adaptive strategy to allocate calibration samples for tighter intervals. Importantly, we characterize parameter regimes defined by the true evaluation score and the LLM judge's sensitivity and specificity in which our LLM-based evaluation yields more reliable estimates than human-only evaluation. Moreover, we show that our framework remains unbiased under distribution shift between the test and calibration datasets, in contrast to existing approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_21140 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How to Correctly Report LLM-as-a-Judge Evaluations Lee, Chungpa Zeng, Thomas Jeong, Jongwon Sohn, Jy-yong Lee, Kangwook Machine Learning Computation and Language Applications Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators. However, imperfect sensitivity and specificity of the LLM judges induce bias in naive evaluation scores. We propose a simple plug-in framework that corrects this bias and enables statistically principled uncertainty quantification. Our framework constructs confidence intervals that account for uncertainty from both the test dataset and a human-labeled calibration dataset. Additionally, it uses an adaptive strategy to allocate calibration samples for tighter intervals. Importantly, we characterize parameter regimes defined by the true evaluation score and the LLM judge's sensitivity and specificity in which our LLM-based evaluation yields more reliable estimates than human-only evaluation. Moreover, we show that our framework remains unbiased under distribution shift between the test and calibration datasets, in contrast to existing approaches. |
| title | How to Correctly Report LLM-as-a-Judge Evaluations |
| topic | Machine Learning Computation and Language Applications |
| url | https://arxiv.org/abs/2511.21140 |