GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911770231177216 |
|---|---|
| author | Du, Cunxiao Jiang, Jing Yuanchen, Xu Wu, Jiawei Yu, Sicheng Li, Yongqi Li, Shenggui Xu, Kai Nie, Liqiang Tu, Zhaopeng You, Yang |
| author_facet | Du, Cunxiao Jiang, Jing Yuanchen, Xu Wu, Jiawei Yu, Sicheng Li, Yongqi Li, Shenggui Xu, Kai Nie, Liqiang Tu, Zhaopeng You, Yang |
| contents | Speculative decoding is a relatively new decoding framework that leverages small and efficient draft models to reduce the latency of LLMs. In this study, we introduce GliDe and CaPE, two low-hassle modifications to vanilla speculative decoding to further improve the decoding speed of a frozen LLM. Specifically, GliDe is a modified draft model architecture that reuses the cached keys and values from the target LLM, while CaPE is a proposal expansion method that uses the draft model's confidence scores to help select additional candidate tokens for verification. Extensive experiments on different benchmarks demonstrate that our proposed GliDe draft model significantly reduces the expected decoding latency. Additional evaluation using walltime reveals that GliDe can accelerate Vicuna models up to 2.17x and further extend the improvement to 2.61x with CaPE. We will release our code, data, and the trained draft models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02082 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding Du, Cunxiao Jiang, Jing Yuanchen, Xu Wu, Jiawei Yu, Sicheng Li, Yongqi Li, Shenggui Xu, Kai Nie, Liqiang Tu, Zhaopeng You, Yang Computation and Language Speculative decoding is a relatively new decoding framework that leverages small and efficient draft models to reduce the latency of LLMs. In this study, we introduce GliDe and CaPE, two low-hassle modifications to vanilla speculative decoding to further improve the decoding speed of a frozen LLM. Specifically, GliDe is a modified draft model architecture that reuses the cached keys and values from the target LLM, while CaPE is a proposal expansion method that uses the draft model's confidence scores to help select additional candidate tokens for verification. Extensive experiments on different benchmarks demonstrate that our proposed GliDe draft model significantly reduces the expected decoding latency. Additional evaluation using walltime reveals that GliDe can accelerate Vicuna models up to 2.17x and further extend the improvement to 2.61x with CaPE. We will release our code, data, and the trained draft models. |
| title | GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.02082 |