Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2506.01096 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915434154950656 |
|---|---|
| author | Liu, Yihao Li, Shuocheng Cao, Lang Xie, Yuhang Zhou, Mengyu Dong, Haoyu Ma, Xiaojun Han, Shi Zhang, Dongmei |
| author_facet | Liu, Yihao Li, Shuocheng Cao, Lang Xie, Yuhang Zhou, Mengyu Dong, Haoyu Ma, Xiaojun Han, Shi Zhang, Dongmei |
| contents | Large language models are increasingly used for complex reasoning tasks where high-quality offline data such as expert-annotated solutions and distilled reasoning traces are often available. However, in environments with sparse rewards, reinforcement learning struggles to sample successful trajectories, leading to inefficient learning. At the same time, these offline trajectories that represent correct reasoning paths are not utilized by standard on-policy reinforcement learning methods. We introduce SuperRL, a unified training framework that adaptively alternates between RL and SFT. Whenever every rollout for a given instance receives zero reward, indicating the absence of a learning signal, SuperRL falls back to SFT on the curated offline data. Extensive experiments across diverse reasoning benchmarks show that SuperRL surpasses vanilla RL by delivering higher sample efficiency, stronger generalization, and improved robustness under sparse rewards. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01096 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning Liu, Yihao Li, Shuocheng Cao, Lang Xie, Yuhang Zhou, Mengyu Dong, Haoyu Ma, Xiaojun Han, Shi Zhang, Dongmei Artificial Intelligence Large language models are increasingly used for complex reasoning tasks where high-quality offline data such as expert-annotated solutions and distilled reasoning traces are often available. However, in environments with sparse rewards, reinforcement learning struggles to sample successful trajectories, leading to inefficient learning. At the same time, these offline trajectories that represent correct reasoning paths are not utilized by standard on-policy reinforcement learning methods. We introduce SuperRL, a unified training framework that adaptively alternates between RL and SFT. Whenever every rollout for a given instance receives zero reward, indicating the absence of a learning signal, SuperRL falls back to SFT on the curated offline data. Extensive experiments across diverse reasoning benchmarks show that SuperRL surpasses vanilla RL by delivering higher sample efficiency, stronger generalization, and improved robustness under sparse rewards. |
| title | SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2506.01096 |