Mitigating Overthinking through Reasoning Shaping
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909835473190912 |
|---|---|
| author | Song, Feifan Wei, Shaohang Gao, Bofei Wang, Yejie Luo, Wen Li, Wei Yao, Linli Xiong, Weimin Chen, Liang Liu, Tianyu Wang, Houfeng |
| author_facet | Song, Feifan Wei, Shaohang Gao, Bofei Wang, Yejie Luo, Wen Li, Wei Yao, Linli Xiong, Weimin Chen, Liang Liu, Tianyu Wang, Houfeng |
| contents | Large reasoning models (LRMs) boosted by Reinforcement Learning from Verifier Reward (RLVR) have shown great power in problem solving, yet they often cause overthinking: excessive, meandering reasoning that inflates computational cost. Prior designs of penalization in RLVR manage to reduce token consumption while often harming model performance, which arises from the oversimplicity of token-level supervision. In this paper, we argue that the granularity of supervision plays a crucial role in balancing efficiency and accuracy, and propose Group Relative Segment Penalization (GRSP), a step-level method to regularize reasoning. Since preliminary analyses show that reasoning segments are strongly correlated with token consumption and model performance, we design a length-aware weighting mechanism across segment clusters. Extensive experiments demonstrate that GRSP achieves superior token efficiency without heavily compromising accuracy, especially the advantages with harder problems. Moreover, GRSP stabilizes RL training and scales effectively across model sizes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09535 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mitigating Overthinking through Reasoning Shaping Song, Feifan Wei, Shaohang Gao, Bofei Wang, Yejie Luo, Wen Li, Wei Yao, Linli Xiong, Weimin Chen, Liang Liu, Tianyu Wang, Houfeng Computation and Language Artificial Intelligence Large reasoning models (LRMs) boosted by Reinforcement Learning from Verifier Reward (RLVR) have shown great power in problem solving, yet they often cause overthinking: excessive, meandering reasoning that inflates computational cost. Prior designs of penalization in RLVR manage to reduce token consumption while often harming model performance, which arises from the oversimplicity of token-level supervision. In this paper, we argue that the granularity of supervision plays a crucial role in balancing efficiency and accuracy, and propose Group Relative Segment Penalization (GRSP), a step-level method to regularize reasoning. Since preliminary analyses show that reasoning segments are strongly correlated with token consumption and model performance, we design a length-aware weighting mechanism across segment clusters. Extensive experiments demonstrate that GRSP achieves superior token efficiency without heavily compromising accuracy, especially the advantages with harder problems. Moreover, GRSP stabilizes RL training and scales effectively across model sizes. |
| title | Mitigating Overthinking through Reasoning Shaping |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.09535 |