Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.13170 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929548883394560 |
|---|---|
| author | Li, Zeping Yang, Xinlong Gao, Ziheng Liu, Ji Li, Guanchen Liu, Zhuang Li, Dong Peng, Jinzhang Tian, Lu Barsoum, Emad |
| author_facet | Li, Zeping Yang, Xinlong Gao, Ziheng Liu, Ji Li, Guanchen Liu, Zhuang Li, Dong Peng, Jinzhang Tian, Lu Barsoum, Emad |
| contents | Large Language Models (LLMs) inherently use autoregressive decoding, which lacks parallelism in inference and results in significantly slow inference speed. While methods such as Medusa constructs parallelized heads, they lack adequate information interaction across different prediction positions. To overcome this limitation, we introduce Amphista, an enhanced speculative decoding framework that builds upon Medusa. Specifically, Amphista models an Auto-embedding Block capable of parallel inference, incorporating bi-directional attention to enable interaction between different drafting heads. Additionally, Amphista integrates Staged Adaptation Layers, which ensure a seamless transition of semantic information from the target model's autoregressive inference to the drafting heads' non-autoregressive inference, effectively achieving paradigm shift and feature fusion. Experimental results on Vicuna models using MT-Bench and Spec-Bench demonstrate that Amphista achieves substantial acceleration while maintaining generation quality. On MT-Bench, Amphista delivers up to 2.75$\times$ speedup over vanilla autoregressive decoding and 1.40$\times$ over Medusa on Vicuna 33B in wall-clock time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13170 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference Li, Zeping Yang, Xinlong Gao, Ziheng Liu, Ji Li, Guanchen Liu, Zhuang Li, Dong Peng, Jinzhang Tian, Lu Barsoum, Emad Artificial Intelligence Computation and Language Large Language Models (LLMs) inherently use autoregressive decoding, which lacks parallelism in inference and results in significantly slow inference speed. While methods such as Medusa constructs parallelized heads, they lack adequate information interaction across different prediction positions. To overcome this limitation, we introduce Amphista, an enhanced speculative decoding framework that builds upon Medusa. Specifically, Amphista models an Auto-embedding Block capable of parallel inference, incorporating bi-directional attention to enable interaction between different drafting heads. Additionally, Amphista integrates Staged Adaptation Layers, which ensure a seamless transition of semantic information from the target model's autoregressive inference to the drafting heads' non-autoregressive inference, effectively achieving paradigm shift and feature fusion. Experimental results on Vicuna models using MT-Bench and Spec-Bench demonstrate that Amphista achieves substantial acceleration while maintaining generation quality. On MT-Bench, Amphista delivers up to 2.75$\times$ speedup over vanilla autoregressive decoding and 1.40$\times$ over Medusa on Vicuna 33B in wall-clock time. |
| title | Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.13170 |