FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929752343838720 |
|---|---|
| author | Zhao, Weilin Pan, Tengyu Han, Xu Zhang, Yudi Sun, Ao Huang, Yuxiang Zhang, Kaihuo Zhao, Weilun Li, Yuxuan Wang, Jianyong Liu, Zhiyuan Sun, Maosong |
| author_facet | Zhao, Weilin Pan, Tengyu Han, Xu Zhang, Yudi Sun, Ao Huang, Yuxiang Zhang, Kaihuo Zhao, Weilun Li, Yuxuan Wang, Jianyong Liu, Zhiyuan Sun, Maosong |
| contents | Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify mechanism to produce multiple tokens per forward pass. While state-of-the-art speculative sampling methods use only a single layer and a language modeling (LM) head as the draft model to achieve impressive layer compression, their efficiency gains are substantially reduced for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens. To address this, we present FR-Spec, a frequency-ranked speculative sampling framework that optimizes draft candidate selection through vocabulary space compression. By constraining the draft search to a frequency-prioritized token subset, our method reduces LM Head computation overhead by 75% while ensuring the equivalence of the final output distribution. Experiments across multiple datasets demonstrate an average of 1.12$\times$ speedup over the state-of-the-art speculative sampling method EAGLE-2. Code available at https://github.com/thunlp/FR-Spec. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14856 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling Zhao, Weilin Pan, Tengyu Han, Xu Zhang, Yudi Sun, Ao Huang, Yuxiang Zhang, Kaihuo Zhao, Weilun Li, Yuxuan Wang, Jianyong Liu, Zhiyuan Sun, Maosong Computation and Language Artificial Intelligence Machine Learning Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify mechanism to produce multiple tokens per forward pass. While state-of-the-art speculative sampling methods use only a single layer and a language modeling (LM) head as the draft model to achieve impressive layer compression, their efficiency gains are substantially reduced for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens. To address this, we present FR-Spec, a frequency-ranked speculative sampling framework that optimizes draft candidate selection through vocabulary space compression. By constraining the draft search to a frequency-prioritized token subset, our method reduces LM Head computation overhead by 75% while ensuring the equivalence of the final output distribution. Experiments across multiple datasets demonstrate an average of 1.12$\times$ speedup over the state-of-the-art speculative sampling method EAGLE-2. Code available at https://github.com/thunlp/FR-Spec. |
| title | FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.14856 |