Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911661858750464 |
|---|---|
| author | Chen, Kejia Zhang, Jiawen Wu, Yihong Gao, Kewei Lou, Jian Feng, Zunlei Song, Mingli Jia, Ruoxi |
| author_facet | Chen, Kejia Zhang, Jiawen Wu, Yihong Gao, Kewei Lou, Jian Feng, Zunlei Song, Mingli Jia, Ruoxi |
| contents | Large reasoning models often reach correct answers through flawed intermediate steps, creating a gap between final accuracy and reasoning reliability. Existing alignment strategies address this with external verifiers or massive sampling, limiting scalability. In this work, we introduce CASPO (Confidence-Aware Step-wise Preference Optimization), a framework that aligns token-level confidence with step-wise logical correctness through iterative Direct Preference Optimization, without training a separate reward model. During inference, we propose Confidence-aware Thought (CaT), which leverages this calibrated confidence to dynamically prune uncertain reasoning branches with negligible O(V) latency. Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency. CASPO scales to Qwen3-8B-Base and surpasses tree-search baselines on AIME'24 and AIME'25 without using reward-model data. We also release a step-wise dataset with confidence annotations to support fine-grained analysis of reasoning reliability. Code is available at https://github.com/Thecommonirin/CASPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_07353 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Confidence-Aware Alignment Makes Reasoning LLMs More Reliable Chen, Kejia Zhang, Jiawen Wu, Yihong Gao, Kewei Lou, Jian Feng, Zunlei Song, Mingli Jia, Ruoxi Artificial Intelligence Large reasoning models often reach correct answers through flawed intermediate steps, creating a gap between final accuracy and reasoning reliability. Existing alignment strategies address this with external verifiers or massive sampling, limiting scalability. In this work, we introduce CASPO (Confidence-Aware Step-wise Preference Optimization), a framework that aligns token-level confidence with step-wise logical correctness through iterative Direct Preference Optimization, without training a separate reward model. During inference, we propose Confidence-aware Thought (CaT), which leverages this calibrated confidence to dynamically prune uncertain reasoning branches with negligible O(V) latency. Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency. CASPO scales to Qwen3-8B-Base and surpasses tree-search baselines on AIME'24 and AIME'25 without using reward-model data. We also release a step-wise dataset with confidence annotations to support fine-grained analysis of reasoning reliability. Code is available at https://github.com/Thecommonirin/CASPO. |
| title | Confidence-Aware Alignment Makes Reasoning LLMs More Reliable |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.07353 |