Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2410.04315 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912304658907136 |
|---|---|
| author | Wang, Peiqi Lam, Barbara D. Liu, Yingcheng Asgari-Targhi, Ameneh Panda, Rameswar Wells, William M. Kapur, Tina Golland, Polina |
| author_facet | Wang, Peiqi Lam, Barbara D. Liu, Yingcheng Asgari-Targhi, Ameneh Panda, Rameswar Wells, William M. Kapur, Tina Golland, Polina |
| contents | We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04315 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Calibrating Expressions of Certainty Wang, Peiqi Lam, Barbara D. Liu, Yingcheng Asgari-Targhi, Ameneh Panda, Rameswar Wells, William M. Kapur, Tina Golland, Polina Computation and Language Machine Learning We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration. |
| title | Calibrating Expressions of Certainty |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.04315 |