Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918328246730752 |
|---|---|
| author | Zhao, Yicong Tsang, King Yeung Vejendla, Harshil Shi, Haizhou Li, Zhuohang Hua, Zhigang Xu, Qi Zhang, Tunyu Wang, Yi Han, Ligong Malin, Bradley A. Wang, Hao |
| author_facet | Zhao, Yicong Tsang, King Yeung Vejendla, Harshil Shi, Haizhou Li, Zhuohang Hua, Zhigang Xu, Qi Zhang, Tunyu Wang, Yi Han, Ligong Malin, Bradley A. Wang, Hao |
| contents | Large Language Models (LLMs) often exhibit misalignment between the quality of their generated responses and the confidence estimates they assign to them. Bayesian treatments, such as marginalizing over a reliable weight posterior or over the space of reasoning traces, provide an effective remedy, but incur substantial computational overhead due to repeated sampling at test time. To enable accurate uncertainty estimation in a single forward pass, we propose a novel distributional distillation framework (Dist2ill) that trains an LLM to produce multiple diverse reasoning paths within one inference pass, while using a lightweight parametric module to approximate empirical confidence scores derived from the sampling distribution. Extensive experiments demonstrate that Dist2ill preserves reasoning diversity and achieves state-of-the-art uncertainty estimation, substantially improving Expected Calibration Error (ECE) and Negative Log-Likelihood (NLL), while remaining computationally efficient. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11731 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models Zhao, Yicong Tsang, King Yeung Vejendla, Harshil Shi, Haizhou Li, Zhuohang Hua, Zhigang Xu, Qi Zhang, Tunyu Wang, Yi Han, Ligong Malin, Bradley A. Wang, Hao Machine Learning Artificial Intelligence Computation and Language Large Language Models (LLMs) often exhibit misalignment between the quality of their generated responses and the confidence estimates they assign to them. Bayesian treatments, such as marginalizing over a reliable weight posterior or over the space of reasoning traces, provide an effective remedy, but incur substantial computational overhead due to repeated sampling at test time. To enable accurate uncertainty estimation in a single forward pass, we propose a novel distributional distillation framework (Dist2ill) that trains an LLM to produce multiple diverse reasoning paths within one inference pass, while using a lightweight parametric module to approximate empirical confidence scores derived from the sampling distribution. Extensive experiments demonstrate that Dist2ill preserves reasoning diversity and achieves state-of-the-art uncertainty estimation, substantially improving Expected Calibration Error (ECE) and Negative Log-Likelihood (NLL), while remaining computationally efficient. |
| title | Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2505.11731 |