Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908790572449792 |
|---|---|
| author | Liu, Haolin Yu, Dian Lu, Sidi Zhou, Yujun Liu, Rui Liang, Zhenwen Mi, Haitao Wei, Chen-Yu Yu, Dong |
| author_facet | Liu, Haolin Yu, Dian Lu, Sidi Zhou, Yujun Liu, Rui Liang, Zhenwen Mi, Haitao Wei, Chen-Yu Yu, Dong |
| contents | Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs). However, most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions. Process reward models (PRMs) offer fine-grained step-level supervision, but their scores are often noisy and difficult to evaluate. As a result, recent PRM benchmarks focus on a more objective capability: detecting the first incorrect step in a reasoning path. However, this evaluation target is misaligned with how PRMs are typically used in RL, where their step-wise scores are treated as raw rewards to maximize. To bridge this gap, we propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL. Given an incorrect rollout, VPPO partitions the trajectory into a verified correct prefix and an erroneous suffix based on the first error, rewarding the former while applying targeted penalties only after the detected mistake. This design yields stable, interpretable learning signals and improves credit assignment. Across multiple reasoning benchmarks, VPPO consistently outperforms sparse-reward RL and prior PRM-guided baselines on both Pass@1 and Pass@K. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18984 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning Liu, Haolin Yu, Dian Lu, Sidi Zhou, Yujun Liu, Rui Liang, Zhenwen Mi, Haitao Wei, Chen-Yu Yu, Dong Machine Learning Computation and Language Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs). However, most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions. Process reward models (PRMs) offer fine-grained step-level supervision, but their scores are often noisy and difficult to evaluate. As a result, recent PRM benchmarks focus on a more objective capability: detecting the first incorrect step in a reasoning path. However, this evaluation target is misaligned with how PRMs are typically used in RL, where their step-wise scores are treated as raw rewards to maximize. To bridge this gap, we propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL. Given an incorrect rollout, VPPO partitions the trajectory into a verified correct prefix and an erroneous suffix based on the first error, rewarding the former while applying targeted penalties only after the detected mistake. This design yields stable, interpretable learning signals and improves credit assignment. Across multiple reasoning benchmarks, VPPO consistently outperforms sparse-reward RL and prior PRM-guided baselines on both Pass@1 and Pass@K. |
| title | Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.18984 |