Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.06419 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917981174366208 |
|---|---|
| author | Neelam, Sanjit Heinlein, Daniel Cvicek, Vaclav Mishra, Akshay Pope, Reiner |
| author_facet | Neelam, Sanjit Heinlein, Daniel Cvicek, Vaclav Mishra, Akshay Pope, Reiner |
| contents | Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3x for small batch sizes. However, increasing throughput and therefore reducing the cost per token requires decoding with large batch sizes. Recent work shows that SD can accelerate decoding with large batch sizes too if the context is sufficiently long and the draft model's KV cache is sparse. We introduce SPIRe, a draft model that combines static sparse attention, pruned initialization, and feedback memory to increase the modeled throughput of speculative decoding by over 100% compared to speculation with a much smaller draft model and by over 35% compared to the strong baseline of sparse self-speculation. Our approach is particularly effective when context lengths vary significantly across requests. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_06419 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SPIRe: Boosting LLM Inference Throughput with Speculative Decoding Neelam, Sanjit Heinlein, Daniel Cvicek, Vaclav Mishra, Akshay Pope, Reiner Machine Learning Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3x for small batch sizes. However, increasing throughput and therefore reducing the cost per token requires decoding with large batch sizes. Recent work shows that SD can accelerate decoding with large batch sizes too if the context is sufficiently long and the draft model's KV cache is sparse. We introduce SPIRe, a draft model that combines static sparse attention, pruned initialization, and feedback memory to increase the modeled throughput of speculative decoding by over 100% compared to speculation with a much smaller draft model and by over 35% compared to the strong baseline of sparse self-speculation. Our approach is particularly effective when context lengths vary significantly across requests. |
| title | SPIRe: Boosting LLM Inference Throughput with Speculative Decoding |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.06419 |