Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.07622 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914461023993856 |
|---|---|
| author | Wang, Ziyi Kasa, Siva Rajesh S, Ankith M Kasa, Santhosh Kumar Zou, Jiaru Negi, Sumit Zhang, Ruqi Jiang, Nan Song, Qifan |
| author_facet | Wang, Ziyi Kasa, Siva Rajesh S, Ankith M Kasa, Santhosh Kumar Zou, Jiaru Negi, Sumit Zhang, Ruqi Jiang, Nan Song, Qifan |
| contents | Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07622 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification Wang, Ziyi Kasa, Siva Rajesh S, Ankith M Kasa, Santhosh Kumar Zou, Jiaru Negi, Sumit Zhang, Ruqi Jiang, Nan Song, Qifan Computation and Language Artificial Intelligence Machine Learning 68T07 I.2.7 Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed. |
| title | DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification |
| topic | Computation and Language Artificial Intelligence Machine Learning 68T07 I.2.7 |
| url | https://arxiv.org/abs/2604.07622 |