Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912072854405120 |
|---|---|
| author | Zhao, Weilin Huang, Yuxiang Han, Xu Xu, Wang Xiao, Chaojun Zhang, Xinrong Fang, Yewei Zhang, Kaihuo Liu, Zhiyuan Sun, Maosong |
| author_facet | Zhao, Weilin Huang, Yuxiang Han, Xu Xu, Wang Xiao, Chaojun Zhang, Xinrong Fang, Yewei Zhang, Kaihuo Liu, Zhiyuan Sun, Maosong |
| contents | Speculative decoding is a widely used method that accelerates the generation process of large language models (LLMs) with no compromise in model performance. It achieves this goal by using an existing smaller model for drafting and then employing the target LLM to verify the draft in a low-cost parallel manner. Under such a drafting-verification framework, drafting efficiency has become a bottleneck in the final speedup of speculative decoding. Therefore, generating longer drafts at less cost can lead to better decoding speedup. To achieve this, we introduce Ouroboros, which can generate draft phrases to parallelize the drafting process and meanwhile lengthen drafts in a training-free manner. The experimental results on various typical text generation tasks show that Ouroboros can achieve speedups of up to $2.8\times$ over speculative decoding and $3.9\times$ over vanilla decoding, without fine-tuning draft and target models. The source code of Ouroboros is available at https://github.com/thunlp/Ouroboros. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13720 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding Zhao, Weilin Huang, Yuxiang Han, Xu Xu, Wang Xiao, Chaojun Zhang, Xinrong Fang, Yewei Zhang, Kaihuo Liu, Zhiyuan Sun, Maosong Computation and Language Speculative decoding is a widely used method that accelerates the generation process of large language models (LLMs) with no compromise in model performance. It achieves this goal by using an existing smaller model for drafting and then employing the target LLM to verify the draft in a low-cost parallel manner. Under such a drafting-verification framework, drafting efficiency has become a bottleneck in the final speedup of speculative decoding. Therefore, generating longer drafts at less cost can lead to better decoding speedup. To achieve this, we introduce Ouroboros, which can generate draft phrases to parallelize the drafting process and meanwhile lengthen drafts in a training-free manner. The experimental results on various typical text generation tasks show that Ouroboros can achieve speedups of up to $2.8\times$ over speculative decoding and $3.9\times$ over vanilla decoding, without fine-tuning draft and target models. The source code of Ouroboros is available at https://github.com/thunlp/Ouroboros. |
| title | Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.13720 |