MineDraft: A Framework for Batch Parallel Speculative Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910280765669376 |
|---|---|
| author | Tang, Zhenwei Verma, Arun Zhou, Zijian Wu, Zhaoxuan Prakash, Alok Rus, Daniela Low, Bryan Kian Hsiang |
| author_facet | Tang, Zhenwei Verma, Arun Zhou, Zijian Wu, Zhaoxuan Prakash, Alok Rus, Daniela Low, Bryan Kian Hsiang |
| contents | Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of MineDraft in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_18016 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MineDraft: A Framework for Batch Parallel Speculative Decoding Tang, Zhenwei Verma, Arun Zhou, Zijian Wu, Zhaoxuan Prakash, Alok Rus, Daniela Low, Bryan Kian Hsiang Computation and Language Artificial Intelligence Distributed, Parallel, and Cluster Computing Machine Learning Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of MineDraft in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems. |
| title | MineDraft: A Framework for Batch Parallel Speculative Decoding |
| topic | Computation and Language Artificial Intelligence Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2603.18016 |