Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918159004467200 |
|---|---|
| author | Xu, Zicheng Wang, Guanchu Zheng, Guangyao Chuang, Yu-Neng Szalay, Alexander Hu, Xia Braverman, Vladimir |
| author_facet | Xu, Zicheng Wang, Guanchu Zheng, Guangyao Chuang, Yu-Neng Szalay, Alexander Hu, Xia Braverman, Vladimir |
| contents | Although Large Language Models (LLMs) perform well in general fields, they exhibit a confidence distortion problem on multi-choice question-answering (MCQA), particularly as the number of answer choices increases. Specifically, on MCQA with many choices, LLMs suffer from under-confidence in correct predictions and over-confidence in incorrect ones, leading to a substantially degraded performance. To solve this problem, we propose Self-ensemble in this work. Our method splits the choices into several groups and ensembles LLM predictions across these groups to reach a final decision. The advantage of Self-ensemble is its plug-and-play nature, where it can be integrated into existing LLM architecture based on a designed attention mask and positional encoding, without requiring labeled datasets for parameter tuning. Experimental results on three LLMs and datasets demonstrate that Self-ensemble comprehensively addresses the confidence distortion problem of LLMs, outperforming standard inference as well as baseline methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01951 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models Xu, Zicheng Wang, Guanchu Zheng, Guangyao Chuang, Yu-Neng Szalay, Alexander Hu, Xia Braverman, Vladimir Computation and Language Machine Learning Although Large Language Models (LLMs) perform well in general fields, they exhibit a confidence distortion problem on multi-choice question-answering (MCQA), particularly as the number of answer choices increases. Specifically, on MCQA with many choices, LLMs suffer from under-confidence in correct predictions and over-confidence in incorrect ones, leading to a substantially degraded performance. To solve this problem, we propose Self-ensemble in this work. Our method splits the choices into several groups and ensembles LLM predictions across these groups to reach a final decision. The advantage of Self-ensemble is its plug-and-play nature, where it can be integrated into existing LLM architecture based on a designed attention mask and positional encoding, without requiring labeled datasets for parameter tuning. Experimental results on three LLMs and datasets demonstrate that Self-ensemble comprehensively addresses the confidence distortion problem of LLMs, outperforming standard inference as well as baseline methods. |
| title | Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.01951 |