SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914254949449728 |
|---|---|
| author | Chang, Chi-Chih Zhu, Siqi Zeng, Zhichen Lin, Haibin You, Jiaxuan Abdelfattah, Mohamed S. Jiang, Ziheng Qian, Xuehai |
| author_facet | Chang, Chi-Chih Zhu, Siqi Zeng, Zhichen Lin, Haibin You, Jiaxuan Abdelfattah, Mohamed S. Jiang, Ziheng Qian, Xuehai |
| contents | We present Speculative Rollout with Tree-Structured Cache (SRT), a simple, model-free approach to accelerate on-policy reinforcement learning (RL) for language models without sacrificing distributional correctness. SRT exploits the empirical similarity of rollouts for the same prompt across training steps by storing previously generated continuations in a per-prompt tree-structured cache. During generation, the current policy uses this tree as the draft model for performing speculative decoding. To keep the cache fresh and improve draft model quality, SRT updates trees online from ongoing rollouts and proactively performs run-ahead generation during idle GPU bubbles. Integrated into standard RL pipelines (\textit{e.g.}, PPO, GRPO and DAPO) and multi-turn settings, SRT consistently reduces generation and step latency and lowers per-token inference cost, achieving up to 2.08x wall-clock time speedup during rollout. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09083 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache Chang, Chi-Chih Zhu, Siqi Zeng, Zhichen Lin, Haibin You, Jiaxuan Abdelfattah, Mohamed S. Jiang, Ziheng Qian, Xuehai Machine Learning We present Speculative Rollout with Tree-Structured Cache (SRT), a simple, model-free approach to accelerate on-policy reinforcement learning (RL) for language models without sacrificing distributional correctness. SRT exploits the empirical similarity of rollouts for the same prompt across training steps by storing previously generated continuations in a per-prompt tree-structured cache. During generation, the current policy uses this tree as the draft model for performing speculative decoding. To keep the cache fresh and improve draft model quality, SRT updates trees online from ongoing rollouts and proactively performs run-ahead generation during idle GPU bubbles. Integrated into standard RL pipelines (\textit{e.g.}, PPO, GRPO and DAPO) and multi-turn settings, SRT consistently reduces generation and step latency and lowers per-token inference cost, achieving up to 2.08x wall-clock time speedup during rollout. |
| title | SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.09083 |