Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914562282881024 |
|---|---|
| author | Dao, Manh Cuong Pham, Quang Hung Nguyen, Phi Le Truong, Thao Nguyen Low, Bryan Kian Hsiang Hoang, Trong Nghia |
| author_facet | Dao, Manh Cuong Pham, Quang Hung Nguyen, Phi Le Truong, Thao Nguyen Low, Bryan Kian Hsiang Hoang, Trong Nghia |
| contents | Uncertainty calibration in pre-trained transformers is critical for their reliable deployment in risk-sensitive applications. Yet, most existing pre-trained transformers do not have a principled mechanism for uncertainty propagation through their feature transformation stack. In this work, we propose a diffusion-inspired reconfiguration of transformers in which each feature transformation block is modeled as a probabilistic mapping. Composing these probabilistic mappings reveals a probability path that mimics the structure of a diffusion process, transporting data mass from the input distribution to the pre-trained feature distribution. This probability path can then be recompiled on a diffusion process with a unified transition model to enable principled propagation of representation uncertainty throughout the pre-trained model's architecture while maintaining its original predictive performance. Empirical results across a variety of vision and language benchmarks demonstrate that our method achieves superior calibration and predictive accuracy compared to existing uncertainty-aware transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08920 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration Dao, Manh Cuong Pham, Quang Hung Nguyen, Phi Le Truong, Thao Nguyen Low, Bryan Kian Hsiang Hoang, Trong Nghia Machine Learning Uncertainty calibration in pre-trained transformers is critical for their reliable deployment in risk-sensitive applications. Yet, most existing pre-trained transformers do not have a principled mechanism for uncertainty propagation through their feature transformation stack. In this work, we propose a diffusion-inspired reconfiguration of transformers in which each feature transformation block is modeled as a probabilistic mapping. Composing these probabilistic mappings reveals a probability path that mimics the structure of a diffusion process, transporting data mass from the input distribution to the pre-trained feature distribution. This probability path can then be recompiled on a diffusion process with a unified transition model to enable principled propagation of representation uncertainty throughout the pre-trained model's architecture while maintaining its original predictive performance. Empirical results across a variety of vision and language benchmarks demonstrate that our method achieves superior calibration and predictive accuracy compared to existing uncertainty-aware transformers. |
| title | Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.08920 |