SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908492875431936 |
|---|---|
| author | Xu, Yuyang Cheng, Yi Ying, Haochao Du, Zhuoyun Hu, Renjun Shi, Xing Lin, Wei Wu, Jian |
| author_facet | Xu, Yuyang Cheng, Yi Ying, Haochao Du, Zhuoyun Hu, Renjun Shi, Xing Lin, Wei Wu, Jian |
| contents | Test-time scaling has proven effective in further enhancing the performance of pretrained Large Language Models (LLMs). However, mainstream post-training methods (i.e., reinforcement learning (RL) with chain-of-thought (CoT) reasoning) often incur substantial computational overhead due to auxiliary models and overthinking. In this paper, we empirically reveal that the incorrect answers partially stem from verbose reasoning processes lacking correct self-fix, where errors accumulate across multiple reasoning steps. To this end, we propose Self-traced Step-wise Preference Optimization (SSPO), a pluggable RL process supervision framework that enables fine-grained optimization of each reasoning step. Specifically, SSPO requires neither auxiliary models nor stepwise manual annotations. Instead, it leverages step-wise preference signals generated by the model itself to guide the optimization process for reasoning compression. Experiments demonstrate that the generated reasoning sequences from SSPO are both accurate and succinct, effectively mitigating overthinking behaviors without compromising model performance across diverse domains and languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_12604 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression Xu, Yuyang Cheng, Yi Ying, Haochao Du, Zhuoyun Hu, Renjun Shi, Xing Lin, Wei Wu, Jian Machine Learning Artificial Intelligence Test-time scaling has proven effective in further enhancing the performance of pretrained Large Language Models (LLMs). However, mainstream post-training methods (i.e., reinforcement learning (RL) with chain-of-thought (CoT) reasoning) often incur substantial computational overhead due to auxiliary models and overthinking. In this paper, we empirically reveal that the incorrect answers partially stem from verbose reasoning processes lacking correct self-fix, where errors accumulate across multiple reasoning steps. To this end, we propose Self-traced Step-wise Preference Optimization (SSPO), a pluggable RL process supervision framework that enables fine-grained optimization of each reasoning step. Specifically, SSPO requires neither auxiliary models nor stepwise manual annotations. Instead, it leverages step-wise preference signals generated by the model itself to guide the optimization process for reasoning compression. Experiments demonstrate that the generated reasoning sequences from SSPO are both accurate and succinct, effectively mitigating overthinking behaviors without compromising model performance across diverse domains and languages. |
| title | SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.12604 |