Accelerating Production LLMs with Combined Token/Embedding Speculators
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917686773022720 |
|---|---|
| author | Wertheimer, Davis Rosenkranz, Joshua Parnell, Thomas Suneja, Sahil Ranganathan, Pavithra Ganti, Raghu Srivatsa, Mudhakar |
| author_facet | Wertheimer, Davis Rosenkranz, Joshua Parnell, Thomas Suneja, Sahil Ranganathan, Pavithra Ganti, Raghu Srivatsa, Mudhakar |
| contents | This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production environment. By conditioning draft predictions on both context vectors and sampled tokens, we can train our speculators to efficiently predict high-quality n-grams, which the base model then accepts or rejects. This allows us to effectively predict multiple tokens per inference forward pass, accelerating wall-clock inference speeds of highly optimized base model implementations by a factor of 2-3x. We explore these initial results and describe next steps for further improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_19124 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Accelerating Production LLMs with Combined Token/Embedding Speculators Wertheimer, Davis Rosenkranz, Joshua Parnell, Thomas Suneja, Sahil Ranganathan, Pavithra Ganti, Raghu Srivatsa, Mudhakar Computation and Language This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production environment. By conditioning draft predictions on both context vectors and sampled tokens, we can train our speculators to efficiently predict high-quality n-grams, which the base model then accepts or rejects. This allows us to effectively predict multiple tokens per inference forward pass, accelerating wall-clock inference speeds of highly optimized base model implementations by a factor of 2-3x. We explore these initial results and describe next steps for further improvements. |
| title | Accelerating Production LLMs with Combined Token/Embedding Speculators |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.19124 |