GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929733973835776 |
|---|---|
| author | Sung, Yoo Yeon Fleisig, Eve Hou, Yu Upadhyay, Ishan Boyd-Graber, Jordan Lee |
| author_facet | Sung, Yoo Yeon Fleisig, Eve Hou, Yu Upadhyay, Ishan Boyd-Graber, Jordan Lee |
| contents | Language models are often miscalibrated, leading to confidently incorrect answers. We introduce GRACE, a benchmark for language model calibration that incorporates comparison with human calibration. GRACE consists of question-answer pairs, in which each question contains a series of clues that gradually become easier, all leading to the same answer; models must answer correctly as early as possible as the clues are revealed. This setting permits granular measurement of model calibration based on how early, accurately, and confidently a model answers. After collecting these questions, we host live human vs. model competitions to gather 1,749 data points on human and model teams' timing, accuracy, and confidence. We propose a metric, CalScore, that uses GRACE to analyze model calibration errors and identify types of model miscalibration that differ from human behavior. We find that although humans are less accurate than models, humans are generally better calibrated. Since state-of-the-art models struggle on GRACE, it effectively evaluates progress on improving model calibration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19684 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration Sung, Yoo Yeon Fleisig, Eve Hou, Yu Upadhyay, Ishan Boyd-Graber, Jordan Lee Computation and Language Language models are often miscalibrated, leading to confidently incorrect answers. We introduce GRACE, a benchmark for language model calibration that incorporates comparison with human calibration. GRACE consists of question-answer pairs, in which each question contains a series of clues that gradually become easier, all leading to the same answer; models must answer correctly as early as possible as the clues are revealed. This setting permits granular measurement of model calibration based on how early, accurately, and confidently a model answers. After collecting these questions, we host live human vs. model competitions to gather 1,749 data points on human and model teams' timing, accuracy, and confidence. We propose a metric, CalScore, that uses GRACE to analyze model calibration errors and identify types of model miscalibration that differ from human behavior. We find that although humans are less accurate than models, humans are generally better calibrated. Since state-of-the-art models struggle on GRACE, it effectively evaluates progress on improving model calibration. |
| title | GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.19684 |