Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.23272 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913568096518144 |
|---|---|
| author | Yang, Qidong Zhu, Weicheng Keslin, Joseph Zanna, Laure Rudner, Tim G. J. Fernandez-Granda, Carlos |
| author_facet | Yang, Qidong Zhu, Weicheng Keslin, Joseph Zanna, Laure Rudner, Tim G. J. Fernandez-Granda, Carlos |
| contents | Probabilistic prediction of sequences from images and other high-dimensional data is a key challenge, particularly in risk-sensitive applications. In these settings, it is often desirable to quantify the uncertainty associated with the prediction (instead of just determining the most likely sequence, as in language modeling). In this paper, we propose a Monte Carlo framework to estimate probabilities and confidence intervals associated with the distribution of a discrete sequence. Our framework uses a Monte Carlo simulator, implemented as an autoregressively trained neural network, to sample sequences conditioned on an image input. We then use these samples to estimate the probabilities and confidence intervals. Experiments on synthetic and real data show that the framework produces accurate discriminative predictions, but can suffer from miscalibration. In order to address this shortcoming, we propose a time-dependent regularization method, which is shown to produce calibrated predictions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23272 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction Yang, Qidong Zhu, Weicheng Keslin, Joseph Zanna, Laure Rudner, Tim G. J. Fernandez-Granda, Carlos Machine Learning Artificial Intelligence Probabilistic prediction of sequences from images and other high-dimensional data is a key challenge, particularly in risk-sensitive applications. In these settings, it is often desirable to quantify the uncertainty associated with the prediction (instead of just determining the most likely sequence, as in language modeling). In this paper, we propose a Monte Carlo framework to estimate probabilities and confidence intervals associated with the distribution of a discrete sequence. Our framework uses a Monte Carlo simulator, implemented as an autoregressively trained neural network, to sample sequences conditioned on an image input. We then use these samples to estimate the probabilities and confidence intervals. Experiments on synthetic and real data show that the framework produces accurate discriminative predictions, but can suffer from miscalibration. In order to address this shortcoming, we propose a time-dependent regularization method, which is shown to produce calibrated predictions. |
| title | A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.23272 |