Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.13634 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910131119194112 |
|---|---|
| author | Zhou, Xuwen Liu, Fangxin Wang, Chao Zheng, Xiao Zheng, Hao He, Min Jiang, Li Guan, Haibing |
| author_facet | Zhou, Xuwen Liu, Fangxin Wang, Chao Zheng, Xiao Zheng, Hao He, Min Jiang, Li Guan, Haibing |
| contents | Speculative decoding accelerates autoregressive generation by letting draft tokens bypass full verification, but conventional frameworks suffer from frequent false rejections, particularly when draft models produce semantically correct but lexically divergent outputs. In this paper, we present Calibrated Speculative Decoding (CSD), a training-free framework that recovers valid tokens discarded by standard verification. Guided by the principle of "Frequency-Guided Candidate Selection and Probability-Guarded Acceptance," CSD incorporates two lightweight modules: Online Correction Memory, which aggregates historical rejections to propose recurring divergence patterns as rescue candidates, and Semantic Consistency Gating, which verifies candidate admissibility using probability ratios instead of exact token matching. Our evaluation across diverse large language models demonstrates that CSD outperforms existing methods, achieving a peak throughput speedup of 2.33x. CSD preserves model accuracy across all tasks while further boosting performance on complex reasoning datasets. These results establish CSD as a highly effective, lightweight solution for practical LLM deployments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13634 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference Zhou, Xuwen Liu, Fangxin Wang, Chao Zheng, Xiao Zheng, Hao He, Min Jiang, Li Guan, Haibing Computation and Language Machine Learning Speculative decoding accelerates autoregressive generation by letting draft tokens bypass full verification, but conventional frameworks suffer from frequent false rejections, particularly when draft models produce semantically correct but lexically divergent outputs. In this paper, we present Calibrated Speculative Decoding (CSD), a training-free framework that recovers valid tokens discarded by standard verification. Guided by the principle of "Frequency-Guided Candidate Selection and Probability-Guarded Acceptance," CSD incorporates two lightweight modules: Online Correction Memory, which aggregates historical rejections to propose recurring divergence patterns as rescue candidates, and Semantic Consistency Gating, which verifies candidate admissibility using probability ratios instead of exact token matching. Our evaluation across diverse large language models demonstrates that CSD outperforms existing methods, achieving a peak throughput speedup of 2.33x. CSD preserves model accuracy across all tasks while further boosting performance on complex reasoning datasets. These results establish CSD as a highly effective, lightweight solution for practical LLM deployments. |
| title | Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2604.13634 |