Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.00263 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911187494502400 |
|---|---|
| author | Li, Zhuohang Li, Xiaowei Huang, Chengyu Li, Guowang Goshvadi, Katayoon Dai, Bo Schuurmans, Dale Zhou, Paul Palangi, Hamid Song, Yiwen Goyal, Palash Kantarcioglu, Murat Malin, Bradley A. Xue, Yuan |
| author_facet | Li, Zhuohang Li, Xiaowei Huang, Chengyu Li, Guowang Goshvadi, Katayoon Dai, Bo Schuurmans, Dale Zhou, Paul Palangi, Hamid Song, Yiwen Goyal, Palash Kantarcioglu, Murat Malin, Bradley A. Xue, Yuan |
| contents | The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or ``autoraters''. However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00263 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Judging with Confidence: Calibrating Autoraters to Preference Distributions Li, Zhuohang Li, Xiaowei Huang, Chengyu Li, Guowang Goshvadi, Katayoon Dai, Bo Schuurmans, Dale Zhou, Paul Palangi, Hamid Song, Yiwen Goyal, Palash Kantarcioglu, Murat Malin, Bradley A. Xue, Yuan Computation and Language The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or ``autoraters''. However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks. |
| title | Judging with Confidence: Calibrating Autoraters to Preference Distributions |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.00263 |