Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.08899 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918157932822528 |
|---|---|
| author | Yin, Junxi Luo, Haisen Li, Zhenyu Liu, Yihua Liu, Dan Li, Zequn Xu, Xiaohang |
| author_facet | Yin, Junxi Luo, Haisen Li, Zhenyu Liu, Yihua Liu, Dan Li, Zequn Xu, Xiaohang |
| contents | While Reinforcement Learning with Verifiable Rewards (RLVR) enhances complex reasoning in LLMs, current methods struggle to balance exploration and exploitation. This leads to critical issues like inaccurate credit assignment for intermediate steps and premature entropy collapse, limiting model performance. To address this, we introduce Attribution-based Contribution to Policy Optimization (ACPO), a phased framework that incorporates a difficulty-aware curriculum. ACPO improves exploration by using trajectory semantic segmentation and an attribution-based representation to dynamically regulate policy entropy, thus mitigating its collapse. Concurrently, it enhances exploitation with a factorized reward system that precisely quantifies the hierarchical contribution of each reasoning step, ensuring accurate credit assignment. Extensive experiments on challenging benchmarks, including AIME, MATH, and AMC, demonstrate that ACPO significantly outperforms existing state-of-the-art approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08899 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning Yin, Junxi Luo, Haisen Li, Zhenyu Liu, Yihua Liu, Dan Li, Zequn Xu, Xiaohang Machine Learning Artificial Intelligence While Reinforcement Learning with Verifiable Rewards (RLVR) enhances complex reasoning in LLMs, current methods struggle to balance exploration and exploitation. This leads to critical issues like inaccurate credit assignment for intermediate steps and premature entropy collapse, limiting model performance. To address this, we introduce Attribution-based Contribution to Policy Optimization (ACPO), a phased framework that incorporates a difficulty-aware curriculum. ACPO improves exploration by using trajectory semantic segmentation and an attribution-based representation to dynamically regulate policy entropy, thus mitigating its collapse. Concurrently, it enhances exploitation with a factorized reward system that precisely quantifies the hierarchical contribution of each reasoning step, ensuring accurate credit assignment. Extensive experiments on challenging benchmarks, including AIME, MATH, and AMC, demonstrate that ACPO significantly outperforms existing state-of-the-art approaches. |
| title | Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.08899 |