Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.11462 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913939300810752 |
|---|---|
| author | Chen, Ziyi Yang, Xiaocong Lin, Jiacheng Sun, Chenkai Chang, Kevin Chen-Chuan Huang, Jie |
| author_facet | Chen, Ziyi Yang, Xiaocong Lin, Jiacheng Sun, Chenkai Chang, Kevin Chen-Chuan Huang, Jie |
| contents | Introduced to enhance the efficiency of large language model (LLM) inference, speculative decoding operates by having a smaller model generate a draft. A larger target model then reviews this draft to align with its output, and any acceptance by the target model results in a reduction of the number of the target model runs, ultimately improving efficiency. However, the drafting process in speculative decoding includes slow autoregressive generation and allocates equal time to generating tokens, irrespective of their importance. These inefficiencies collectively contribute to the suboptimal performance of speculative decoding. To further improve LLM inference, we introduce Cascade Speculative Drafting (CS Drafting), a speculative execution algorithm that incorporates two types of cascades. The Vertical Cascade eliminates autoregressive generation from neural models, while the Horizontal Cascade optimizes time allocation in drafting for improved efficiency. Combining both cascades, CS Drafting achieves greater speedup compared to the baselines in our experiments, while preserving the same output distribution as the target model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_11462 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Cascade Speculative Drafting for Even Faster LLM Inference Chen, Ziyi Yang, Xiaocong Lin, Jiacheng Sun, Chenkai Chang, Kevin Chen-Chuan Huang, Jie Machine Learning Computation and Language Introduced to enhance the efficiency of large language model (LLM) inference, speculative decoding operates by having a smaller model generate a draft. A larger target model then reviews this draft to align with its output, and any acceptance by the target model results in a reduction of the number of the target model runs, ultimately improving efficiency. However, the drafting process in speculative decoding includes slow autoregressive generation and allocates equal time to generating tokens, irrespective of their importance. These inefficiencies collectively contribute to the suboptimal performance of speculative decoding. To further improve LLM inference, we introduce Cascade Speculative Drafting (CS Drafting), a speculative execution algorithm that incorporates two types of cascades. The Vertical Cascade eliminates autoregressive generation from neural models, while the Horizontal Cascade optimizes time allocation in drafting for improved efficiency. Combining both cascades, CS Drafting achieves greater speedup compared to the baselines in our experiments, while preserving the same output distribution as the target model. |
| title | Cascade Speculative Drafting for Even Faster LLM Inference |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2312.11462 |