Uncertainty Quantification of Large Language Models using Approximate Bayesian Computation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911172288053248 |
|---|---|
| author | Sharma, Mridul Patel, Adeetya Souza, Zaneta D' Rahimi, Samira Abbasgholizadeh Reddy, Siva Madathil, Sreenath |
| author_facet | Sharma, Mridul Patel, Adeetya Souza, Zaneta D' Rahimi, Samira Abbasgholizadeh Reddy, Siva Madathil, Sreenath |
| contents | Despite their widespread applications, Large Language Models (LLMs) often struggle to express uncertainty, posing a challenge for reliable deployment in high stakes and safety critical domains like clinical diagnostics. Existing standard baseline methods such as model logits and elicited probabilities produce overconfident and poorly calibrated estimates. In this work, we propose Approximate Bayesian Computation (ABC), a likelihood-free Bayesian inference, based approach that treats LLMs as a stochastic simulator to infer posterior distributions over predictive probabilities. We evaluate our ABC approach on two clinically relevant benchmarks: a synthetic oral lesion diagnosis dataset and the publicly available GretelAI symptom-to-diagnosis dataset. Compared to standard baselines, our approach improves accuracy by up to 46.9\%, reduces Brier scores by 74.4\%, and enhances calibration as measured by Expected Calibration Error (ECE) and predictive entropy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19375 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Uncertainty Quantification of Large Language Models using Approximate Bayesian Computation Sharma, Mridul Patel, Adeetya Souza, Zaneta D' Rahimi, Samira Abbasgholizadeh Reddy, Siva Madathil, Sreenath Machine Learning Artificial Intelligence Despite their widespread applications, Large Language Models (LLMs) often struggle to express uncertainty, posing a challenge for reliable deployment in high stakes and safety critical domains like clinical diagnostics. Existing standard baseline methods such as model logits and elicited probabilities produce overconfident and poorly calibrated estimates. In this work, we propose Approximate Bayesian Computation (ABC), a likelihood-free Bayesian inference, based approach that treats LLMs as a stochastic simulator to infer posterior distributions over predictive probabilities. We evaluate our ABC approach on two clinically relevant benchmarks: a synthetic oral lesion diagnosis dataset and the publicly available GretelAI symptom-to-diagnosis dataset. Compared to standard baselines, our approach improves accuracy by up to 46.9\%, reduces Brier scores by 74.4\%, and enhances calibration as measured by Expected Calibration Error (ECE) and predictive entropy. |
| title | Uncertainty Quantification of Large Language Models using Approximate Bayesian Computation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.19375 |