Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.15672 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914482325815296 |
|---|---|
| author | Emara, Yahya da Costa, Mauricio Barba Chang, Chi-Chih Freer, Cameron Vieira, Tim Cotterell, Ryan Abdelfattah, Mohamed S. |
| author_facet | Emara, Yahya da Costa, Mauricio Barba Chang, Chi-Chih Freer, Cameron Vieira, Tim Cotterell, Ryan Abdelfattah, Mohamed S. |
| contents | Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sampling. Because rejection truncates the draft block at the first error, throughput degrades when draft and target diverge. Rather than rejecting draft tokens outright, we propose to reweight them. To this end, we introduce sequential Monte Carlo speculative decoding (SMC-SD), which replaces token-level rejection with importance-weighted resampling over a population of draft particles. SMC-SD is a principled approximate inference scheme that trades exactness for additional speed, while preserving theoretical bounds on its per-step approximation error. Because LLM inference is memory bandwidth-bound, the arithmetic needed to draft particles and to score them in parallel comes nearly for free -- SMC-SD uses idle compute to turn verification into a vectorized, fixed-size operation with no rollback. Empirically, SMC-SD achieves 2.36x speed-up over speculative decoding and a 5.2x speed-up over autoregressive decoding, while remaining within 3% of the target model's accuracy on reasoning, instruction-following, and coding benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_15672 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Faster LLM Inference via Sequential Monte Carlo Emara, Yahya da Costa, Mauricio Barba Chang, Chi-Chih Freer, Cameron Vieira, Tim Cotterell, Ryan Abdelfattah, Mohamed S. Machine Learning Computation and Language Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sampling. Because rejection truncates the draft block at the first error, throughput degrades when draft and target diverge. Rather than rejecting draft tokens outright, we propose to reweight them. To this end, we introduce sequential Monte Carlo speculative decoding (SMC-SD), which replaces token-level rejection with importance-weighted resampling over a population of draft particles. SMC-SD is a principled approximate inference scheme that trades exactness for additional speed, while preserving theoretical bounds on its per-step approximation error. Because LLM inference is memory bandwidth-bound, the arithmetic needed to draft particles and to score them in parallel comes nearly for free -- SMC-SD uses idle compute to turn verification into a vectorized, fixed-size operation with no rollback. Empirically, SMC-SD achieves 2.36x speed-up over speculative decoding and a 5.2x speed-up over autoregressive decoding, while remaining within 3% of the target model's accuracy on reasoning, instruction-following, and coding benchmarks. |
| title | Faster LLM Inference via Sequential Monte Carlo |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2604.15672 |