How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909088615497728 |
|---|---|
| author | Hashimoto, Kazuma Naim, Iftekhar Raman, Karthik |
| author_facet | Hashimoto, Kazuma Naim, Iftekhar Raman, Karthik |
| contents | Sequence labeling is a core task in text understanding for IE/IR systems. Text generation models have increasingly become the go-to solution for such tasks (e.g., entity extraction and dialog slot filling). While most research has focused on the labeling accuracy, a key aspect -- of vital practical importance -- has slipped through the cracks: understanding model confidence. More specifically, we lack a principled understanding of how to reliably gauge the confidence of a model in its predictions for each labeled span. This paper aims to provide some empirical insights on estimating model confidence for generative sequence labeling. Most notably, we find that simply using the decoder's output probabilities \textbf{is not} the best in realizing well-calibrated confidence estimates. As verified over six public datasets of different tasks, we show that our proposed approach -- which leverages statistics from top-$k$ predictions by a beam search -- significantly reduces calibration errors of the predictions of a generative sequence labeling model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2212_10767 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling? Hashimoto, Kazuma Naim, Iftekhar Raman, Karthik Computation and Language Sequence labeling is a core task in text understanding for IE/IR systems. Text generation models have increasingly become the go-to solution for such tasks (e.g., entity extraction and dialog slot filling). While most research has focused on the labeling accuracy, a key aspect -- of vital practical importance -- has slipped through the cracks: understanding model confidence. More specifically, we lack a principled understanding of how to reliably gauge the confidence of a model in its predictions for each labeled span. This paper aims to provide some empirical insights on estimating model confidence for generative sequence labeling. Most notably, we find that simply using the decoder's output probabilities \textbf{is not} the best in realizing well-calibrated confidence estimates. As verified over six public datasets of different tasks, we show that our proposed approach -- which leverages statistics from top-$k$ predictions by a beam search -- significantly reduces calibration errors of the predictions of a generative sequence labeling model. |
| title | How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2212.10767 |