Calibrated Confidence Estimation for Tabular Question Answering
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908962500116480 |
|---|---|
| author | Voss, Lukas |
| author_facet | Voss, Lukas |
| contents | Large language models (LLMs) are increasingly deployed for tabular question
answering, yet calibration on structured data is largely unstudied. This
paper presents the first systematic comparison of five confidence estimation
methods across five frontier LLMs and two tabular QA benchmarks. All models
are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported
for textual QA). A consistent self-evaluation versus perturbation dichotomy
replicates across both benchmarks and all four fully-covered models:
self-evaluation methods (verbalized, P(True)) achieve AUROC 0.42-0.76, while
perturbation methods (semantic entropy, self-consistency, and our
Multi-Format Agreement) achieve AUROC 0.78-0.86. Per-model paired bootstrap
tests reject the null at p<0.001 after Holm-Bonferroni correction, and a
3-seed check on GPT-4o-mini gives a per-seed standard deviation of only
0.006. The paper proposes Multi-Format Agreement (MFA), which exploits the
lossless and deterministic serialization variation unique to structured data
(Markdown, HTML, JSON, CSV) to estimate confidence at 20% lower API cost
than sampling baselines. MFA reduces ECE by 44-63%, generalizes across all
four models on TableBench (mean AUROC 0.80), and combines complementarily
with sampling: an MFA + self-consistency ensemble lifts AUROC from 0.74 to
0.82. A secondary contribution, structure-aware recalibration, improves
AUROC by +10 percentage points over standard post-hoc methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_12491 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Calibrated Confidence Estimation for Tabular Question Answering Voss, Lukas Computation and Language I.2.7; I.2.6 Large language models (LLMs) are increasingly deployed for tabular question answering, yet calibration on structured data is largely unstudied. This paper presents the first systematic comparison of five confidence estimation methods across five frontier LLMs and two tabular QA benchmarks. All models are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported for textual QA). A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models: self-evaluation methods (verbalized, P(True)) achieve AUROC 0.42-0.76, while perturbation methods (semantic entropy, self-consistency, and our Multi-Format Agreement) achieve AUROC 0.78-0.86. Per-model paired bootstrap tests reject the null at p<0.001 after Holm-Bonferroni correction, and a 3-seed check on GPT-4o-mini gives a per-seed standard deviation of only 0.006. The paper proposes Multi-Format Agreement (MFA), which exploits the lossless and deterministic serialization variation unique to structured data (Markdown, HTML, JSON, CSV) to estimate confidence at 20% lower API cost than sampling baselines. MFA reduces ECE by 44-63%, generalizes across all four models on TableBench (mean AUROC 0.80), and combines complementarily with sampling: an MFA + self-consistency ensemble lifts AUROC from 0.74 to 0.82. A secondary contribution, structure-aware recalibration, improves AUROC by +10 percentage points over standard post-hoc methods. |
| title | Calibrated Confidence Estimation for Tabular Question Answering |
| topic | Computation and Language I.2.7; I.2.6 |
| url | https://arxiv.org/abs/2604.12491 |