Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911296248610816 |
|---|---|
| author | Zhao, Yilong Tang, Jiaming Zhu, Kan Ye, Zihao Chang, Chi-Chih Lin, Chaofan Park, Jongseok Xiao, Guangxuan Abdelfattah, Mohamed S. Gao, Mingyu Kasikci, Baris Han, Song Stoica, Ion |
| author_facet | Zhao, Yilong Tang, Jiaming Zhu, Kan Ye, Zihao Chang, Chi-Chih Lin, Chaofan Park, Jongseok Xiao, Guangxuan Abdelfattah, Mohamed S. Gao, Mingyu Kasikci, Baris Han, Song Stoica, Ion |
| contents | Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation shifts the inference bottleneck from compute-bound to memory-bound. To generate each token, the model applies full attention to all previously generated tokens, requiring memory access to an increasingly large KV-Cache. Consequently, longer generations demand more memory access for every step, leading to substantial pressure on memory bandwidth.
To address this, we introduce SparseSpec, a speculative decoding framework that reuses the same model as the draft and target models (i.e., self-speculation). SparseSpec features a novel sparse attention mechanism, PillarAttn, as the draft model, which accurately selects critical tokens via elegantly reusing information from the verification stage. Furthermore, SparseSpec co-designs self-speculation with three system innovations: (1) a unified scheduler to batch token drafting and verification, (2) delayed verification for CPU/GPU overlap, and (3) dynamic KV-Cache management to maximize memory utilization. Across various models and datasets, SparseSpec outperforms state-of-the-art solutions, with an up to 2.13x throughput speedup. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_01278 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding Zhao, Yilong Tang, Jiaming Zhu, Kan Ye, Zihao Chang, Chi-Chih Lin, Chaofan Park, Jongseok Xiao, Guangxuan Abdelfattah, Mohamed S. Gao, Mingyu Kasikci, Baris Han, Song Stoica, Ion Machine Learning Artificial Intelligence Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation shifts the inference bottleneck from compute-bound to memory-bound. To generate each token, the model applies full attention to all previously generated tokens, requiring memory access to an increasingly large KV-Cache. Consequently, longer generations demand more memory access for every step, leading to substantial pressure on memory bandwidth. To address this, we introduce SparseSpec, a speculative decoding framework that reuses the same model as the draft and target models (i.e., self-speculation). SparseSpec features a novel sparse attention mechanism, PillarAttn, as the draft model, which accurately selects critical tokens via elegantly reusing information from the verification stage. Furthermore, SparseSpec co-designs self-speculation with three system innovations: (1) a unified scheduler to batch token drafting and verification, (2) delayed verification for CPU/GPU overlap, and (3) dynamic KV-Cache management to maximize memory utilization. Across various models and datasets, SparseSpec outperforms state-of-the-art solutions, with an up to 2.13x throughput speedup. |
| title | Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2512.01278 |