Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913066389602304 |
|---|---|
| author | Vazhentsev, Artem Rvanova, Lyudmila Lazichny, Ivan Panchenko, Alexander Panov, Maxim Baldwin, Timothy Shelmanov, Artem |
| author_facet | Vazhentsev, Artem Rvanova, Lyudmila Lazichny, Ivan Panchenko, Alexander Panov, Maxim Baldwin, Timothy Shelmanov, Artem |
| contents | Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs). To date, information-based and consistency-based UQ have been the dominant UQ methods for text generation via LLMs. Density-based methods, despite being very effective for UQ in text classification with encoder-based models, have not been very successful with generative LLMs. In this work, we adapt Mahalanobis Distance (MD) - a well-established UQ technique in classification tasks - for text generation and introduce a new supervised UQ method. Our method extracts token embeddings from multiple layers of LLMs, computes MD scores for each token, and uses linear regression trained on these features to provide robust uncertainty scores. Through extensive experiments on eleven datasets, we demonstrate that our approach substantially improves over existing UQ methods, providing accurate and computationally efficient uncertainty scores for both sequence-level selective generation and claim-level fact-checking tasks. Our method also exhibits strong generalization to out-of-domain data, making it suitable for a wide range of LLM-based applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14427 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models Vazhentsev, Artem Rvanova, Lyudmila Lazichny, Ivan Panchenko, Alexander Panov, Maxim Baldwin, Timothy Shelmanov, Artem Computation and Language Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs). To date, information-based and consistency-based UQ have been the dominant UQ methods for text generation via LLMs. Density-based methods, despite being very effective for UQ in text classification with encoder-based models, have not been very successful with generative LLMs. In this work, we adapt Mahalanobis Distance (MD) - a well-established UQ technique in classification tasks - for text generation and introduce a new supervised UQ method. Our method extracts token embeddings from multiple layers of LLMs, computes MD scores for each token, and uses linear regression trained on these features to provide robust uncertainty scores. Through extensive experiments on eleven datasets, we demonstrate that our approach substantially improves over existing UQ methods, providing accurate and computationally efficient uncertainty scores for both sequence-level selective generation and claim-level fact-checking tasks. Our method also exhibits strong generalization to out-of-domain data, making it suitable for a wide range of LLM-based applications. |
| title | Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.14427 |