Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.05636 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909487054454784 |
|---|---|
| author | Christopher, Jacob K Bartoldson, Brian R Ben-Nun, Tal Cardei, Michael Kailkhura, Bhavya Fioretto, Ferdinando |
| author_facet | Christopher, Jacob K Bartoldson, Brian R Ben-Nun, Tal Cardei, Michael Kailkhura, Bhavya Fioretto, Ferdinando |
| contents | Speculative decoding has emerged as a widely adopted method to accelerate large language model inference without sacrificing the quality of the model outputs. While this technique has facilitated notable speed improvements by enabling parallel sequence verification, its efficiency remains inherently limited by the reliance on incremental token generation in existing draft models. To overcome this limitation, this paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences. This allows parallelization of both the drafting and verification steps, providing significant speedups to the inference process. Our proposed approach, $\textit{Speculative Diffusion Decoding (SpecDiff)}$, is validated on standard language generation benchmarks and empirically demonstrated to provide up to 7.2x speedups over standard generation processes and up to 1.75x speedups over existing speculative decoding approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_05636 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion Christopher, Jacob K Bartoldson, Brian R Ben-Nun, Tal Cardei, Michael Kailkhura, Bhavya Fioretto, Ferdinando Computation and Language Machine Learning Speculative decoding has emerged as a widely adopted method to accelerate large language model inference without sacrificing the quality of the model outputs. While this technique has facilitated notable speed improvements by enabling parallel sequence verification, its efficiency remains inherently limited by the reliance on incremental token generation in existing draft models. To overcome this limitation, this paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences. This allows parallelization of both the drafting and verification steps, providing significant speedups to the inference process. Our proposed approach, $\textit{Speculative Diffusion Decoding (SpecDiff)}$, is validated on standard language generation benchmarks and empirically demonstrated to provide up to 7.2x speedups over standard generation processes and up to 1.75x speedups over existing speculative decoding approaches. |
| title | Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2408.05636 |