Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917133227655168 |
|---|---|
| author | Vashurin, Roman Goloburda, Maiya Ilina, Albina Rubashevskii, Aleksandr Nakov, Preslav Shelmanov, Artem Panov, Maxim |
| author_facet | Vashurin, Roman Goloburda, Maiya Ilina, Albina Rubashevskii, Aleksandr Nakov, Preslav Shelmanov, Artem Panov, Maxim |
| contents | Uncertainty quantification (UQ) methods for Large Language Models (LLMs) encompass a variety of approaches, with two major types being particularly prominent: information-based, which focus on model confidence expressed as token probabilities, and consistency-based, which assess the semantic relationship between multiple outputs generated using repeated sampling. Several recent methods have combined these two approaches to boost UQ performance. However, they sometimes fail to outperform much simpler baseline methods. Our work discusses the fundamental approach to constructing uncertainty measures that directly links uncertainty with the minimum Bayes risks achieved by LLM decoding. Building on these findings, we propose a novel approach to integrating model confidence with output consistency, resulting in a family of efficient and robust UQ methods. Our investigation reveals distinctive characteristics of LLMs as probabilistic models, which help to explain why these UQ methods underperform in certain tasks. Based on these findings, we propose a new way of synthesizing model confidence and output consistency, leading to a family of efficient and robust UQ methods. We evaluate our approach across various tasks such as question answering, abstractive summarization, and machine translation, demonstrating sizable improvements over state-of-the-art UQ approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_04964 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency Vashurin, Roman Goloburda, Maiya Ilina, Albina Rubashevskii, Aleksandr Nakov, Preslav Shelmanov, Artem Panov, Maxim Computation and Language Uncertainty quantification (UQ) methods for Large Language Models (LLMs) encompass a variety of approaches, with two major types being particularly prominent: information-based, which focus on model confidence expressed as token probabilities, and consistency-based, which assess the semantic relationship between multiple outputs generated using repeated sampling. Several recent methods have combined these two approaches to boost UQ performance. However, they sometimes fail to outperform much simpler baseline methods. Our work discusses the fundamental approach to constructing uncertainty measures that directly links uncertainty with the minimum Bayes risks achieved by LLM decoding. Building on these findings, we propose a novel approach to integrating model confidence with output consistency, resulting in a family of efficient and robust UQ methods. Our investigation reveals distinctive characteristics of LLMs as probabilistic models, which help to explain why these UQ methods underperform in certain tasks. Based on these findings, we propose a new way of synthesizing model confidence and output consistency, leading to a family of efficient and robust UQ methods. We evaluate our approach across various tasks such as question answering, abstractive summarization, and machine translation, demonstrating sizable improvements over state-of-the-art UQ approaches. |
| title | Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.04964 |