Calibrating Expressions of Certainty
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912304658907136 |
|---|---|
| author | Wang, Peiqi Lam, Barbara D. Liu, Yingcheng Asgari-Targhi, Ameneh Panda, Rameswar Wells, William M. Kapur, Tina Golland, Polina |
| author_facet | Wang, Peiqi Lam, Barbara D. Liu, Yingcheng Asgari-Targhi, Ameneh Panda, Rameswar Wells, William M. Kapur, Tina Golland, Polina |
| contents | We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04315 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Calibrating Expressions of Certainty Wang, Peiqi Lam, Barbara D. Liu, Yingcheng Asgari-Targhi, Ameneh Panda, Rameswar Wells, William M. Kapur, Tina Golland, Polina Computation and Language Machine Learning We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration. |
| title | Calibrating Expressions of Certainty |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.04315 |