Supervised Optimism Correction: Be Confident When LLMs Are Sure
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918040911740928 |
|---|---|
| author | Zhang, Junjie Yang, Rushuai Liu, Shunyu Lin, Ting-En Huang, Fei Chen, Yi Li, Yongbin Tao, Dacheng |
| author_facet | Zhang, Junjie Yang, Rushuai Liu, Shunyu Lin, Ting-En Huang, Fei Chen, Yi Li, Yongbin Tao, Dacheng |
| contents | In this work, we establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning under the token-level Markov decision process, revealing that large language models indeed learn an implicit $Q$-function for inference. Through this theoretical lens, we demonstrate that the widely used beam search method suffers from unacceptable over-optimism, where inference errors are inevitably amplified due to inflated $Q$-value estimations of suboptimal steps. To address this limitation, we propose Supervised Optimism Correction(SOC), which introduces a simple yet effective auxiliary loss for token-level $Q$-value estimations during supervised fine-tuning. Specifically, the auxiliary loss employs implicit value regularization to boost model confidence in expert-demonstrated responses, thereby suppressing over-optimism toward insufficiently supervised responses. Extensive experiments on mathematical reasoning benchmarks, including GSM8K, MATH, and GAOKAO, showcase the superiority of the proposed SOC with beam search across a series of open-source models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07527 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Supervised Optimism Correction: Be Confident When LLMs Are Sure Zhang, Junjie Yang, Rushuai Liu, Shunyu Lin, Ting-En Huang, Fei Chen, Yi Li, Yongbin Tao, Dacheng Computation and Language In this work, we establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning under the token-level Markov decision process, revealing that large language models indeed learn an implicit $Q$-function for inference. Through this theoretical lens, we demonstrate that the widely used beam search method suffers from unacceptable over-optimism, where inference errors are inevitably amplified due to inflated $Q$-value estimations of suboptimal steps. To address this limitation, we propose Supervised Optimism Correction(SOC), which introduces a simple yet effective auxiliary loss for token-level $Q$-value estimations during supervised fine-tuning. Specifically, the auxiliary loss employs implicit value regularization to boost model confidence in expert-demonstrated responses, thereby suppressing over-optimism toward insufficiently supervised responses. Extensive experiments on mathematical reasoning benchmarks, including GSM8K, MATH, and GAOKAO, showcase the superiority of the proposed SOC with beam search across a series of open-source models. |
| title | Supervised Optimism Correction: Be Confident When LLMs Are Sure |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.07527 |