Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2606.00869 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913176369496064 |
|---|---|
| author | Li, Weitao Zhou, Hao Lei, Xuanyu Meng, Fandong Liu, Yuanhang Ren, Jingyi Wang, Ante Wang, Xiaolong Zhang, Yuanchi Luo, Fuwen Yang, Guangwen Gan, Lin Ma, Weizhi Liu, Yang |
| author_facet | Li, Weitao Zhou, Hao Lei, Xuanyu Meng, Fandong Liu, Yuanhang Ren, Jingyi Wang, Ante Wang, Xiaolong Zhang, Yuanchi Luo, Fuwen Yang, Guangwen Gan, Lin Ma, Weizhi Liu, Yang |
| contents | Reinforcement learning with verifiable rewards (RLVR) has become central to LLM reasoning, but its outcome-level rewards can make models more willing to give confident answers when evidence or reasoning is unreliable. Existing SFT or RL methods mainly teach LLMs to refuse or express uncertainty at the response level, which can overfit abstention behavior rather than improve reasoning reliability. To address this limitation, we propose Cognitive Pairwise Training (CPT), a cognitive mid-training alignment stage that turns pairwise comparisons over reasoning traces into a reusable alignment signal. By learning to distinguish trustworthy from flawed reasoning, CPT encourages the model to internalize a reasoning-quality discrimination boundary rather than memorize surface refusal patterns. Across five model scales and three model families, CPT improves the reasoning--metacognition trade-off. At 14B, CPT+RL outperforms the standard SFT+RL pipeline by +2.2 math-average points and +5.2 abstention-F1 points. Further analyses show that CPT improves trace quality and exhibits strong robustness and scalability across evaluation and training settings. Code and models are released at https://github.com/Tsinghua-dhy/CPT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00869 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Enhancing LLM Metacognition via Cognitive Pairwise Training Li, Weitao Zhou, Hao Lei, Xuanyu Meng, Fandong Liu, Yuanhang Ren, Jingyi Wang, Ante Wang, Xiaolong Zhang, Yuanchi Luo, Fuwen Yang, Guangwen Gan, Lin Ma, Weizhi Liu, Yang Machine Learning Reinforcement learning with verifiable rewards (RLVR) has become central to LLM reasoning, but its outcome-level rewards can make models more willing to give confident answers when evidence or reasoning is unreliable. Existing SFT or RL methods mainly teach LLMs to refuse or express uncertainty at the response level, which can overfit abstention behavior rather than improve reasoning reliability. To address this limitation, we propose Cognitive Pairwise Training (CPT), a cognitive mid-training alignment stage that turns pairwise comparisons over reasoning traces into a reusable alignment signal. By learning to distinguish trustworthy from flawed reasoning, CPT encourages the model to internalize a reasoning-quality discrimination boundary rather than memorize surface refusal patterns. Across five model scales and three model families, CPT improves the reasoning--metacognition trade-off. At 14B, CPT+RL outperforms the standard SFT+RL pipeline by +2.2 math-average points and +5.2 abstention-F1 points. Further analyses show that CPT improves trace quality and exhibits strong robustness and scalability across evaluation and training settings. Code and models are released at https://github.com/Tsinghua-dhy/CPT. |
| title | Enhancing LLM Metacognition via Cognitive Pairwise Training |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2606.00869 |