APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917258950868992 |
|---|---|
| author | Chang, Kaiyan Zhu, Chenwei Luo, Yingfeng Huo, Yifu Wang, Chenglong Liu, Xiaoqian He, Qiaozhi Xiao, Tong Yu, Zhengtao Zhu, Jingbo |
| author_facet | Chang, Kaiyan Zhu, Chenwei Luo, Yingfeng Huo, Yifu Wang, Chenglong Liu, Xiaoqian He, Qiaozhi Xiao, Tong Yu, Zhengtao Zhu, Jingbo |
| contents | Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs) but introduces a critical side-effect known as Overthinking. We conduct a preliminary study to rethink this phenomenon from a fine-grained perspective. We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process. We formally define this specific position where the answer first stabilizes as the Reasoning Anchor. By analyzing pre- and post-anchor reasoning behaviors, we uncover the structural redundancy fixed in LRMs: the meaningless repetitive verification after deriving the first complete answer, which we term the Answer-Stable Tail (AST). Motivated by this observation, we propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST. Leveraging the policy optimization algorithm suitable for length penalties, our APR models achieved the performance-efficiency Pareto frontier at 1.5B and 7B scales averaged across five mathematical reasoning datasets while requiring substantially fewer computational resources for RL training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00760 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards Chang, Kaiyan Zhu, Chenwei Luo, Yingfeng Huo, Yifu Wang, Chenglong Liu, Xiaoqian He, Qiaozhi Xiao, Tong Yu, Zhengtao Zhu, Jingbo Computation and Language Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs) but introduces a critical side-effect known as Overthinking. We conduct a preliminary study to rethink this phenomenon from a fine-grained perspective. We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process. We formally define this specific position where the answer first stabilizes as the Reasoning Anchor. By analyzing pre- and post-anchor reasoning behaviors, we uncover the structural redundancy fixed in LRMs: the meaningless repetitive verification after deriving the first complete answer, which we term the Answer-Stable Tail (AST). Motivated by this observation, we propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST. Leveraging the policy optimization algorithm suitable for length penalties, our APR models achieved the performance-efficiency Pareto frontier at 1.5B and 7B scales averaged across five mathematical reasoning datasets while requiring substantially fewer computational resources for RL training. |
| title | APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.00760 |