TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908384614154240 |
|---|---|
| author | Wu, Zhaoxuan Zhou, Zijian Verma, Arun Prakash, Alok Rus, Daniela Low, Bryan Kian Hsiang |
| author_facet | Wu, Zhaoxuan Zhou, Zijian Verma, Arun Prakash, Alok Rus, Daniela Low, Bryan Kian Hsiang |
| contents | We propose TETRIS, a novel method that optimizes the total throughput of batch speculative decoding in multi-request settings. Unlike existing methods that optimize for a single request or a group of requests as a whole, TETRIS actively selects the most promising draft tokens (for every request in a batch) to be accepted when verified in parallel, resulting in fewer rejected tokens and hence less wasted computing resources. Such an effective resource utilization to achieve fast inference in large language models (LLMs) is especially important to service providers with limited inference capacity. Compared to baseline speculative decoding, TETRIS yields a consistently higher acceptance rate and more effective utilization of the limited inference capacity. We show theoretically and empirically that TETRIS outperforms baseline speculative decoding and existing methods that dynamically select draft tokens, leading to a more efficient batch inference in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15197 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding Wu, Zhaoxuan Zhou, Zijian Verma, Arun Prakash, Alok Rus, Daniela Low, Bryan Kian Hsiang Computation and Language Artificial Intelligence We propose TETRIS, a novel method that optimizes the total throughput of batch speculative decoding in multi-request settings. Unlike existing methods that optimize for a single request or a group of requests as a whole, TETRIS actively selects the most promising draft tokens (for every request in a batch) to be accepted when verified in parallel, resulting in fewer rejected tokens and hence less wasted computing resources. Such an effective resource utilization to achieve fast inference in large language models (LLMs) is especially important to service providers with limited inference capacity. Compared to baseline speculative decoding, TETRIS yields a consistently higher acceptance rate and more effective utilization of the limited inference capacity. We show theoretically and empirically that TETRIS outperforms baseline speculative decoding and existing methods that dynamically select draft tokens, leading to a more efficient batch inference in LLMs. |
| title | TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.15197 |