ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908620288950272 |
|---|---|
| author | Chen, Qiaoling Liu, Zijun Sun, Peng Li, Shenggui Wang, Guoteng Liu, Ziming Wen, Yonggang Feng, Siyuan Zhang, Tianwei |
| author_facet | Chen, Qiaoling Liu, Zijun Sun, Peng Li, Shenggui Wang, Guoteng Liu, Ziming Wen, Yonggang Feng, Siyuan Zhang, Tianwei |
| contents | Adapting large language models (LLMs) via reinforcement learning (RL) is often bottlenecked by the generation stage, which can consume over 75\% of the training time. Speculative decoding (SD) accelerates autoregressive generation in serving systems, but its behavior under RL training remains largely unexplored. We identify three critical gaps that hinder the naive integration of SD into RL systems: diminishing speedups at large batch sizes, drafter staleness under continual actor updates, and drafter-induced policy degradation.
To address these gaps, we present ReSpec, a system that adapts SD to RL through three complementary mechanisms: dynamically tuning SD configurations, evolving the drafter via knowledge distillation, and weighting updates by rollout rewards. On Qwen models (3B--14B), ReSpec achieves up to 4.5x speedup while preserving reward convergence and training stability, providing a practical solution for efficient RL-based LLM adaptation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_26475 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems Chen, Qiaoling Liu, Zijun Sun, Peng Li, Shenggui Wang, Guoteng Liu, Ziming Wen, Yonggang Feng, Siyuan Zhang, Tianwei Machine Learning Distributed, Parallel, and Cluster Computing Adapting large language models (LLMs) via reinforcement learning (RL) is often bottlenecked by the generation stage, which can consume over 75\% of the training time. Speculative decoding (SD) accelerates autoregressive generation in serving systems, but its behavior under RL training remains largely unexplored. We identify three critical gaps that hinder the naive integration of SD into RL systems: diminishing speedups at large batch sizes, drafter staleness under continual actor updates, and drafter-induced policy degradation. To address these gaps, we present ReSpec, a system that adapts SD to RL through three complementary mechanisms: dynamically tuning SD configurations, evolving the drafter via knowledge distillation, and weighting updates by rollout rewards. On Qwen models (3B--14B), ReSpec achieves up to 4.5x speedup while preserving reward convergence and training stability, providing a practical solution for efficient RL-based LLM adaptation. |
| title | ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2510.26475 |