Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908534194569216 |
|---|---|
| author | Wang, Jikai Tian, Zhenxu Li, Juntao Xia, Qingrong Duan, Xinyu Wang, Zhefeng Huai, Baoxing Zhang, Min |
| author_facet | Wang, Jikai Tian, Zhenxu Li, Juntao Xia, Qingrong Duan, Xinyu Wang, Zhefeng Huai, Baoxing Zhang, Min |
| contents | Recent works have revealed the great potential of speculative decoding in accelerating the autoregressive generation process of large language models. The success of these methods relies on the alignment between draft candidates and the sampled outputs of the target model. Existing methods mainly achieve draft-target alignment with training-based methods, e.g., EAGLE, Medusa, involving considerable training costs. In this paper, we present a training-free alignment-augmented speculative decoding algorithm. We propose alignment sampling, which leverages output distribution obtained in the prefilling phase to provide more aligned draft candidates. To further benefit from high-quality but non-aligned draft candidates, we also introduce a simple yet effective flexible verification strategy. Through an adaptive probability threshold, our approach can improve generation accuracy while further improving inference efficiency. Experiments on 8 datasets (including question answering, summarization and code completion tasks) show that our approach increases the average generation score by 3.3 points for the LLaMA3 model. Our method achieves a mean acceptance length up to 2.39 and speed up generation by 2.23. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13204 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification Wang, Jikai Tian, Zhenxu Li, Juntao Xia, Qingrong Duan, Xinyu Wang, Zhefeng Huai, Baoxing Zhang, Min Computation and Language Recent works have revealed the great potential of speculative decoding in accelerating the autoregressive generation process of large language models. The success of these methods relies on the alignment between draft candidates and the sampled outputs of the target model. Existing methods mainly achieve draft-target alignment with training-based methods, e.g., EAGLE, Medusa, involving considerable training costs. In this paper, we present a training-free alignment-augmented speculative decoding algorithm. We propose alignment sampling, which leverages output distribution obtained in the prefilling phase to provide more aligned draft candidates. To further benefit from high-quality but non-aligned draft candidates, we also introduce a simple yet effective flexible verification strategy. Through an adaptive probability threshold, our approach can improve generation accuracy while further improving inference efficiency. Experiments on 8 datasets (including question answering, summarization and code completion tasks) show that our approach increases the average generation score by 3.3 points for the LLaMA3 model. Our method achieves a mean acceptance length up to 2.39 and speed up generation by 2.23. |
| title | Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.13204 |