Bridging Human and LLM Judgments: Understanding and Narrowing the Gap
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909939373441024 |
|---|---|
| author | Polo, Felipe Maia Wang, Xinhe Yurochkin, Mikhail Xu, Gongjun Banerjee, Moulinath Sun, Yuekai |
| author_facet | Polo, Felipe Maia Wang, Xinhe Yurochkin, Mikhail Xu, Gongjun Banerjee, Moulinath Sun, Yuekai |
| contents | Large language models are increasingly used as judges (LLM-as-a-judge) to evaluate model outputs at scale, but their assessments often diverge systematically from human judgments. We present Bridge, a unified statistical framework that explicitly bridges human and LLM evaluations under both absolute scoring and pairwise comparison paradigms. Bridge posits a latent human preference score for each prompt-response pair and models LLM deviations as linear transformations of covariates that capture sources of discrepancies. This offers a simple and principled framework for refining LLM ratings and characterizing systematic discrepancies between humans and LLMs. We provide an efficient fitting algorithm with asymptotic guarantees for statistical inference. Using six LLM judges and two benchmarks (BigGen Bench and Chatbot Arena), Bridge achieves higher agreement with human ratings (accuracy, calibration, and KL divergence) and exposes systematic human-LLM gaps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_12792 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bridging Human and LLM Judgments: Understanding and Narrowing the Gap Polo, Felipe Maia Wang, Xinhe Yurochkin, Mikhail Xu, Gongjun Banerjee, Moulinath Sun, Yuekai Machine Learning Artificial Intelligence Computation and Language Large language models are increasingly used as judges (LLM-as-a-judge) to evaluate model outputs at scale, but their assessments often diverge systematically from human judgments. We present Bridge, a unified statistical framework that explicitly bridges human and LLM evaluations under both absolute scoring and pairwise comparison paradigms. Bridge posits a latent human preference score for each prompt-response pair and models LLM deviations as linear transformations of covariates that capture sources of discrepancies. This offers a simple and principled framework for refining LLM ratings and characterizing systematic discrepancies between humans and LLMs. We provide an efficient fitting algorithm with asymptotic guarantees for statistical inference. Using six LLM judges and two benchmarks (BigGen Bench and Chatbot Arena), Bridge achieves higher agreement with human ratings (accuracy, calibration, and KL divergence) and exposes systematic human-LLM gaps. |
| title | Bridging Human and LLM Judgments: Understanding and Narrowing the Gap |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2508.12792 |