MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labels
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917930624614400 |
|---|---|
| author | Liu, Xiaoou Lin, Zhen Da, Longchao Chen, Chacha Trivedi, Shubhendu Wei, Hua |
| author_facet | Liu, Xiaoou Lin, Zhen Da, Longchao Chen, Chacha Trivedi, Shubhendu Wei, Hua |
| contents | Large Language Models (LLMs) require robust confidence estimation, particularly in critical domains like healthcare and law where unreliable outputs can lead to significant consequences. Despite much recent work in confidence estimation, current evaluation frameworks rely on correctness functions -- various heuristics that are often noisy, expensive, and possibly introduce systematic biases. These methodological weaknesses tend to distort evaluation metrics and thus the comparative ranking of confidence measures. We introduce MCQA-Eval, an evaluation framework for assessing confidence measures in Natural Language Generation (NLG) that eliminates dependence on an explicit correctness function by leveraging gold-standard correctness labels from multiple-choice datasets. MCQA-Eval enables systematic comparison of both internal state-based white-box (e.g. logit-based) and consistency-based black-box confidence measures, providing a unified evaluation methodology across different approaches. Through extensive experiments on multiple LLMs and widely used QA datasets, we report that MCQA-Eval provides efficient and more reliable assessments of confidence estimation methods than existing approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14268 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labels Liu, Xiaoou Lin, Zhen Da, Longchao Chen, Chacha Trivedi, Shubhendu Wei, Hua Computation and Language Artificial Intelligence Large Language Models (LLMs) require robust confidence estimation, particularly in critical domains like healthcare and law where unreliable outputs can lead to significant consequences. Despite much recent work in confidence estimation, current evaluation frameworks rely on correctness functions -- various heuristics that are often noisy, expensive, and possibly introduce systematic biases. These methodological weaknesses tend to distort evaluation metrics and thus the comparative ranking of confidence measures. We introduce MCQA-Eval, an evaluation framework for assessing confidence measures in Natural Language Generation (NLG) that eliminates dependence on an explicit correctness function by leveraging gold-standard correctness labels from multiple-choice datasets. MCQA-Eval enables systematic comparison of both internal state-based white-box (e.g. logit-based) and consistency-based black-box confidence measures, providing a unified evaluation methodology across different approaches. Through extensive experiments on multiple LLMs and widely used QA datasets, we report that MCQA-Eval provides efficient and more reliable assessments of confidence estimation methods than existing approaches. |
| title | MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labels |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.14268 |