Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.20252 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911102803116032 |
|---|---|
| author | Fei, Xiang Wang, Siqi Wei, Shu Nie, Yuxiang Shi, Wei Feng, Hao Feng, Chao Huang, Can |
| author_facet | Fei, Xiang Wang, Siqi Wei, Shu Nie, Yuxiang Shi, Wei Feng, Hao Feng, Chao Huang, Can |
| contents | Current language model training paradigms typically terminate learning upon reaching the end-of-sequence (<eos>) token, overlooking the potential learning opportunities in the post-completion space. We propose Post-Completion Learning (PCL), a novel training framework that systematically utilizes the sequence space after model output completion, to enhance both the reasoning and self-evaluation abilities. PCL enables models to continue generating self-assessments and reward predictions during training, while maintaining efficient inference by stopping at the completion point.
To fully utilize this post-completion space, we design a white-box reinforcement learning method: let the model evaluate the output content according to the reward rules, then calculate and align the score with the reward functions for supervision. We implement dual-track SFT to optimize both reasoning and evaluation capabilities, and mixed it with RL training to achieve multi-objective hybrid optimization.
Experimental results on different datasets and models demonstrate consistent improvements over traditional SFT and RL methods. Our method provides a new technical path for language model training that enhances output quality while preserving deployment efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_20252 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Post-Completion Learning for Language Models Fei, Xiang Wang, Siqi Wei, Shu Nie, Yuxiang Shi, Wei Feng, Hao Feng, Chao Huang, Can Computation and Language Artificial Intelligence Current language model training paradigms typically terminate learning upon reaching the end-of-sequence (<eos>) token, overlooking the potential learning opportunities in the post-completion space. We propose Post-Completion Learning (PCL), a novel training framework that systematically utilizes the sequence space after model output completion, to enhance both the reasoning and self-evaluation abilities. PCL enables models to continue generating self-assessments and reward predictions during training, while maintaining efficient inference by stopping at the completion point. To fully utilize this post-completion space, we design a white-box reinforcement learning method: let the model evaluate the output content according to the reward rules, then calculate and align the score with the reward functions for supervision. We implement dual-track SFT to optimize both reasoning and evaluation capabilities, and mixed it with RL training to achieve multi-objective hybrid optimization. Experimental results on different datasets and models demonstrate consistent improvements over traditional SFT and RL methods. Our method provides a new technical path for language model training that enhances output quality while preserving deployment efficiency. |
| title | Post-Completion Learning for Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2507.20252 |