Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908750728658944 |
|---|---|
| author | Li, Weiyue Zhao, Minda Dong, Weixuan Cai, Jiahui Wei, Yuze Pocress, Michael Li, Yi Yuan, Wanyan Wang, Xiaoyue Hou, Ruoyu Lou, Kaiyuan Zeng, Wenqi Yang, Yutong Du, Yilun Wang, Mengyu |
| author_facet | Li, Weiyue Zhao, Minda Dong, Weixuan Cai, Jiahui Wei, Yuze Pocress, Michael Li, Yi Yuan, Wanyan Wang, Xiaoyue Hou, Ruoyu Lou, Kaiyuan Zeng, Wenqi Yang, Yutong Du, Yilun Wang, Mengyu |
| contents | Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03444 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale Li, Weiyue Zhao, Minda Dong, Weixuan Cai, Jiahui Wei, Yuze Pocress, Michael Li, Yi Yuan, Wanyan Wang, Xiaoyue Hou, Ruoyu Lou, Kaiyuan Zeng, Wenqi Yang, Yutong Du, Yilun Wang, Mengyu Computation and Language Artificial Intelligence Human-Computer Interaction Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols. |
| title | Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2601.03444 |