EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915543043276800 |
|---|---|
| author | Wang, Chen Wei, Lai Zhang, Yanzhi Shao, Chenyang Dan, Zedong Huang, Weiran Zhang, Yuzhi Wang, Yue |
| author_facet | Wang, Chen Wei, Lai Zhang, Yanzhi Shao, Chenyang Dan, Zedong Huang, Weiran Zhang, Yuzhi Wang, Yue |
| contents | Recent advances in reinforcement learning (RL) have significantly enhanced the reasoning capabilities of large language models (LLMs). Group Relative Policy Optimization (GRPO), a lightweight variant of Proximal Policy Optimization (PPO), improves efficiency but suffers from limited exploration and training instability, limiting its effectiveness on complex reasoning tasks. To address these challenges, we introduce EFRame, an Exploration-Filter-Replay framework that augments GRPO across three dimensions: additional rollouts enable deeper and more targeted exploration, online filtering removes low-quality samples to stabilize gradients and accelerate training, and experience replay amplifies rare yet informative trajectories for stable convergence. This unified framework establishes a principled training cycle that balances exploration, efficiency, and stability. Experiments on diverse reasoning benchmarks demonstrate that EFRame achieves consistent gains, including a 37.9\% relative improvement on Geometry3K over GRPO. EFRame further supports fine-grained sample categorization and precise entropy control, highlighting it as a robust solution for advancing deeper reasoning in LLMs. Our code is available at https://github.com/597358816/EFRame. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_22200 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework Wang, Chen Wei, Lai Zhang, Yanzhi Shao, Chenyang Dan, Zedong Huang, Weiran Zhang, Yuzhi Wang, Yue Machine Learning Artificial Intelligence Recent advances in reinforcement learning (RL) have significantly enhanced the reasoning capabilities of large language models (LLMs). Group Relative Policy Optimization (GRPO), a lightweight variant of Proximal Policy Optimization (PPO), improves efficiency but suffers from limited exploration and training instability, limiting its effectiveness on complex reasoning tasks. To address these challenges, we introduce EFRame, an Exploration-Filter-Replay framework that augments GRPO across three dimensions: additional rollouts enable deeper and more targeted exploration, online filtering removes low-quality samples to stabilize gradients and accelerate training, and experience replay amplifies rare yet informative trajectories for stable convergence. This unified framework establishes a principled training cycle that balances exploration, efficiency, and stability. Experiments on diverse reasoning benchmarks demonstrate that EFRame achieves consistent gains, including a 37.9\% relative improvement on Geometry3K over GRPO. EFRame further supports fine-grained sample categorization and precise entropy control, highlighting it as a robust solution for advancing deeper reasoning in LLMs. Our code is available at https://github.com/597358816/EFRame. |
| title | EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.22200 |