REST: Retrieval-Based Speculative Decoding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911827813728256 |
|---|---|
| author | He, Zhenyu Zhong, Zexuan Cai, Tianle Lee, Jason D. He, Di |
| author_facet | He, Zhenyu Zhong, Zexuan Cai, Tianle Lee, Jason D. He, Di |
| contents | We introduce Retrieval-Based Speculative Decoding (REST), a novel algorithm designed to speed up language model generation. The key insight driving the development of REST is the observation that the process of text generation often includes certain common phases and patterns. Unlike previous methods that rely on a draft language model for speculative decoding, REST harnesses the power of retrieval to generate draft tokens. This method draws from the reservoir of existing knowledge, retrieving and employing relevant tokens based on the current context. Its plug-and-play nature allows for seamless integration and acceleration of any language models, all without necessitating additional training. When benchmarked on 7B and 13B language models in a single-batch setting, REST achieves a significant speedup of 1.62X to 2.36X on code or text generation. The code of REST is available at https://github.com/FasterDecoding/REST. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_08252 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | REST: Retrieval-Based Speculative Decoding He, Zhenyu Zhong, Zexuan Cai, Tianle Lee, Jason D. He, Di Computation and Language Artificial Intelligence Information Retrieval Machine Learning We introduce Retrieval-Based Speculative Decoding (REST), a novel algorithm designed to speed up language model generation. The key insight driving the development of REST is the observation that the process of text generation often includes certain common phases and patterns. Unlike previous methods that rely on a draft language model for speculative decoding, REST harnesses the power of retrieval to generate draft tokens. This method draws from the reservoir of existing knowledge, retrieving and employing relevant tokens based on the current context. Its plug-and-play nature allows for seamless integration and acceleration of any language models, all without necessitating additional training. When benchmarked on 7B and 13B language models in a single-batch setting, REST achieves a significant speedup of 1.62X to 2.36X on code or text generation. The code of REST is available at https://github.com/FasterDecoding/REST. |
| title | REST: Retrieval-Based Speculative Decoding |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2311.08252 |