Block Verification Accelerates Speculative Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909574428098560 |
|---|---|
| author | Sun, Ziteng Mendlovic, Uri Leviathan, Yaniv Aharoni, Asaf Ro, Jae Hun Beirami, Ahmad Suresh, Ananda Theertha |
| author_facet | Sun, Ziteng Mendlovic, Uri Leviathan, Yaniv Aharoni, Asaf Ro, Jae Hun Beirami, Ahmad Suresh, Ananda Theertha |
| contents | Speculative decoding is an effective method for lossless acceleration of large language models during inference. It uses a fast model to draft a block of tokens which are then verified in parallel by the target model, and provides a guarantee that the output is distributed identically to a sample from the target model. In prior works, draft verification is performed independently token-by-token. Surprisingly, we show that this approach is not optimal. We propose Block Verification, a simple draft verification algorithm that verifies the entire block jointly and provides additional wall-clock speedup. We prove that the proposed mechanism is optimal in the expected number of tokens produced each iteration and specifically is never worse than the standard token-level verification. Empirically, block verification provides modest but consistent wall-clock speedups over the standard token verification algorithm of 5%-8% in a range of tasks and datasets. Given that block verification does not increase code complexity, maintains the strong lossless guarantee of the standard speculative decoding verification algorithm, cannot deteriorate performance, and, in fact, consistently improves it, it can be used as a good default in speculative decoding implementations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_10444 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Block Verification Accelerates Speculative Decoding Sun, Ziteng Mendlovic, Uri Leviathan, Yaniv Aharoni, Asaf Ro, Jae Hun Beirami, Ahmad Suresh, Ananda Theertha Machine Learning Computation and Language Data Structures and Algorithms Information Theory Speculative decoding is an effective method for lossless acceleration of large language models during inference. It uses a fast model to draft a block of tokens which are then verified in parallel by the target model, and provides a guarantee that the output is distributed identically to a sample from the target model. In prior works, draft verification is performed independently token-by-token. Surprisingly, we show that this approach is not optimal. We propose Block Verification, a simple draft verification algorithm that verifies the entire block jointly and provides additional wall-clock speedup. We prove that the proposed mechanism is optimal in the expected number of tokens produced each iteration and specifically is never worse than the standard token-level verification. Empirically, block verification provides modest but consistent wall-clock speedups over the standard token verification algorithm of 5%-8% in a range of tasks and datasets. Given that block verification does not increase code complexity, maintains the strong lossless guarantee of the standard speculative decoding verification algorithm, cannot deteriorate performance, and, in fact, consistently improves it, it can be used as a good default in speculative decoding implementations. |
| title | Block Verification Accelerates Speculative Decoding |
| topic | Machine Learning Computation and Language Data Structures and Algorithms Information Theory |
| url | https://arxiv.org/abs/2403.10444 |