Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917498485473280 |
|---|---|
| author | Zhu, Tianshu Zhang, Wenyu Zuo, Xiaoying Tian, Lun Zhao, Haotian Zeng, Yucheng Gu, Jingnan Dong, Daxiang Wu, Jianmin Yin, Dawei Shen, Dou |
| author_facet | Zhu, Tianshu Zhang, Wenyu Zuo, Xiaoying Tian, Lun Zhao, Haotian Zeng, Yucheng Gu, Jingnan Dong, Daxiang Wu, Jianmin Yin, Dawei Shen, Dou |
| contents | Agentic reinforcement learning (RL) for software engineering spends much of its compute on stateful trajectories whose grouped binary rewards are highly skewed and weakly contrastive. We frame this as pass-rate control and show that the binary reward-side signal is strongest near a 50% rollout pass rate under four criteria: reward entropy, group-filtering survival, leave-one-out (RLOO) advantage energy under Group Relative Policy Optimization (GRPO), and success-failure pair count. We propose Prefix Sampling (PS), which replays self-generated trajectory prefixes to steer skewed groups toward this regime: successful prefixes give mostly failing groups a head start, while failing prefixes handicap mostly passing groups. Replayed states are reconstructed through the existing rollout path, and replayed tokens are masked from the loss so optimization applies only to current-policy continuations. On SWE-bench Verified, PS reaches the baseline high-score regime within evaluation variability while delivering 2.01x and 1.55x end-to-end wall-clock speedups on Qwen3-14B and Qwen3-32B; the 14B peak improves from 0.274 to 0.295. AIME 2025 experiments on 4B and 8B show the same pass-rate-control pattern, and 4B ablations attribute gains to replay, bidirectional coverage, and adaptive control. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_05112 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime Zhu, Tianshu Zhang, Wenyu Zuo, Xiaoying Tian, Lun Zhao, Haotian Zeng, Yucheng Gu, Jingnan Dong, Daxiang Wu, Jianmin Yin, Dawei Shen, Dou Machine Learning Agentic reinforcement learning (RL) for software engineering spends much of its compute on stateful trajectories whose grouped binary rewards are highly skewed and weakly contrastive. We frame this as pass-rate control and show that the binary reward-side signal is strongest near a 50% rollout pass rate under four criteria: reward entropy, group-filtering survival, leave-one-out (RLOO) advantage energy under Group Relative Policy Optimization (GRPO), and success-failure pair count. We propose Prefix Sampling (PS), which replays self-generated trajectory prefixes to steer skewed groups toward this regime: successful prefixes give mostly failing groups a head start, while failing prefixes handicap mostly passing groups. Replayed states are reconstructed through the existing rollout path, and replayed tokens are masked from the loss so optimization applies only to current-policy continuations. On SWE-bench Verified, PS reaches the baseline high-score regime within evaluation variability while delivering 2.01x and 1.55x end-to-end wall-clock speedups on Qwen3-14B and Qwen3-32B; the 14B peak improves from 0.274 to 0.295. AIME 2025 experiments on 4B and 8B show the same pass-rate-control pattern, and 4B ablations attribute gains to replay, bidirectional coverage, and adaptive control. |
| title | Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.05112 |