Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.03708 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915772467511296 |
|---|---|
| author | Dong, Ximing Wang, Shaowei Lin, Dayi Chen, Boyuan Hassan, Ahmed E. |
| author_facet | Dong, Ximing Wang, Shaowei Lin, Dayi Chen, Boyuan Hassan, Ahmed E. |
| contents | Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large Reasoning Models (LRMs), which generate lengthy chains of thought. While speculative decoding accelerates inference by drafting and verifying multiple tokens in parallel, existing methods operate at the token level and ignore semantic equivalence (i.e., different token sequences expressing the same meaning), leading to inefficient rejections. We propose SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens. SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings. Experiments on four benchmarks show that SemanticSpec achieves up to 2.7x speedup on DeepSeekR1-32B and 2.1x on QwQ-32B, consistently outperforming token-level and sequence-level baselines in both efficiency and effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_03708 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States Dong, Ximing Wang, Shaowei Lin, Dayi Chen, Boyuan Hassan, Ahmed E. Computation and Language Performance Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large Reasoning Models (LRMs), which generate lengthy chains of thought. While speculative decoding accelerates inference by drafting and verifying multiple tokens in parallel, existing methods operate at the token level and ignore semantic equivalence (i.e., different token sequences expressing the same meaning), leading to inefficient rejections. We propose SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens. SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings. Experiments on four benchmarks show that SemanticSpec achieves up to 2.7x speedup on DeepSeekR1-32B and 2.1x on QwQ-32B, consistently outperforming token-level and sequence-level baselines in both efficiency and effectiveness. |
| title | Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States |
| topic | Computation and Language Performance |
| url | https://arxiv.org/abs/2602.03708 |