Identifying Influential N-grams in Confidence Calibration via Regression Analysis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913010378866688 |
|---|---|
| author | Ozaki, Shintaro Hashimoto, Wataru Kamigaito, Hidetaka Hayashi, Katsuhiko Watanabe, Taro |
| author_facet | Ozaki, Shintaro Hashimoto, Wataru Kamigaito, Hidetaka Hayashi, Katsuhiko Watanabe, Taro |
| contents | While large language models (LLMs) improve performance by explicit reasoning, their responses are often overconfident, even though they include linguistic expressions demonstrating uncertainty. In this work, we identify what linguistic expressions are related to confidence by applying the regression method. Specifically, we predict confidence of those linguistic expressions in the reasoning parts of LLMs as the dependent variables and analyze the relationship between a specific $n$-gram and confidence. Across multiple models and QA benchmarks, we show that LLMs remain overconfident when reasoning is involved and attribute this behavior to specific linguistic information. Interestingly, several of the extracted expressions coincide with cue phrases intentionally inserted on test-time scaling to improve reasoning performance. Through our test on causality and verification that the extracted linguistic information truly affects confidence, we reveal that confidence calibration is possible by simply suppressing those overconfident expressions without drops in performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05757 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Identifying Influential N-grams in Confidence Calibration via Regression Analysis Ozaki, Shintaro Hashimoto, Wataru Kamigaito, Hidetaka Hayashi, Katsuhiko Watanabe, Taro Computation and Language While large language models (LLMs) improve performance by explicit reasoning, their responses are often overconfident, even though they include linguistic expressions demonstrating uncertainty. In this work, we identify what linguistic expressions are related to confidence by applying the regression method. Specifically, we predict confidence of those linguistic expressions in the reasoning parts of LLMs as the dependent variables and analyze the relationship between a specific $n$-gram and confidence. Across multiple models and QA benchmarks, we show that LLMs remain overconfident when reasoning is involved and attribute this behavior to specific linguistic information. Interestingly, several of the extracted expressions coincide with cue phrases intentionally inserted on test-time scaling to improve reasoning performance. Through our test on causality and verification that the extracted linguistic information truly affects confidence, we reveal that confidence calibration is possible by simply suppressing those overconfident expressions without drops in performance. |
| title | Identifying Influential N-grams in Confidence Calibration via Regression Analysis |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.05757 |