Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914538434068480 |
|---|---|
| author | Fadeeva, Ekaterina Goloburda, Maiya Rubashevskii, Aleksandr Vashurin, Roman Shelmanov, Artem Nakov, Preslav Sachan, Mrinmaya Panov, Maxim |
| author_facet | Fadeeva, Ekaterina Goloburda, Maiya Rubashevskii, Aleksandr Vashurin, Roman Shelmanov, Artem Nakov, Preslav Sachan, Mrinmaya Panov, Maxim |
| contents | Consistency-based methods have emerged as an effective approach to uncertainty quantification (UQ) in large language models. These methods typically rely on several generations obtained via multinomial sampling, measuring their agreement level. However, in short-form QA, multinomial sampling is prone to producing duplicates due to peaked distributions, and its stochasticity introduces considerable variance in uncertainty estimates across runs. We introduce a new family of methods that employ beam search to generate candidates for consistency-based UQ, yielding improved performance and reduced variance compared to multinomial sampling. We also provide a theoretical lower bound on the beam set probability mass under which beam search achieves a smaller error than multinomial sampling. We empirically evaluate our approach on six QA datasets and find that its consistent improvements over multinomial sampling lead to state-of-the-art UQ performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_09538 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search Fadeeva, Ekaterina Goloburda, Maiya Rubashevskii, Aleksandr Vashurin, Roman Shelmanov, Artem Nakov, Preslav Sachan, Mrinmaya Panov, Maxim Machine Learning Computation and Language Consistency-based methods have emerged as an effective approach to uncertainty quantification (UQ) in large language models. These methods typically rely on several generations obtained via multinomial sampling, measuring their agreement level. However, in short-form QA, multinomial sampling is prone to producing duplicates due to peaked distributions, and its stochasticity introduces considerable variance in uncertainty estimates across runs. We introduce a new family of methods that employ beam search to generate candidates for consistency-based UQ, yielding improved performance and reduced variance compared to multinomial sampling. We also provide a theoretical lower bound on the beam set probability mass under which beam search achieves a smaller error than multinomial sampling. We empirically evaluate our approach on six QA datasets and find that its consistent improvements over multinomial sampling lead to state-of-the-art UQ performance. |
| title | Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2512.09538 |