DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917049866911744 |
|---|---|
| author | Yang, Mingyu Choi, Jae-Young Moon, Kihyo Jang, Minsung Jeon, Eunjoo |
| author_facet | Yang, Mingyu Choi, Jae-Young Moon, Kihyo Jang, Minsung Jeon, Eunjoo |
| contents | Speculative decoding accelerates large language model inference, but its reliance on a fixed speculation length is suboptimal in large-batch serving environments with diverse requests. This paper explores a new direction for dynamic adaptation by investigating a novel class of post-hoc, diagnostic signals. We propose Dynamic Speculative Decoding Engine (DSDE), a training-free framework built on two primary components: (1) a predictive signal based on the variance of the Kullback-Leibler (KLD) divergence, which diagnoses the generation's regional stability, and (2) an adaptive speculation length cap to mitigate the straggler problem in per-sequence decoding. Experiments demonstrate the potential of using KLD-based stability signals for dynamic adaptation. An algorithm guided by these signals achieves end-to-end latency competitive with leading baselines and exhibits superior robustness across diverse workloads. This robustness is particularly valuable in challenging low-acceptance-rate regimes, where the proposed signal maintains its diagnostic utility. Collectively, these findings validate post-hoc signals as a valuable component for building more robust and intelligent LLM inference systems, and highlight a promising direction for future research on dynamic speculation length adaptation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_01083 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving Yang, Mingyu Choi, Jae-Young Moon, Kihyo Jang, Minsung Jeon, Eunjoo Distributed, Parallel, and Cluster Computing Artificial Intelligence Information Theory I.2.7; C.2.4 Speculative decoding accelerates large language model inference, but its reliance on a fixed speculation length is suboptimal in large-batch serving environments with diverse requests. This paper explores a new direction for dynamic adaptation by investigating a novel class of post-hoc, diagnostic signals. We propose Dynamic Speculative Decoding Engine (DSDE), a training-free framework built on two primary components: (1) a predictive signal based on the variance of the Kullback-Leibler (KLD) divergence, which diagnoses the generation's regional stability, and (2) an adaptive speculation length cap to mitigate the straggler problem in per-sequence decoding. Experiments demonstrate the potential of using KLD-based stability signals for dynamic adaptation. An algorithm guided by these signals achieves end-to-end latency competitive with leading baselines and exhibits superior robustness across diverse workloads. This robustness is particularly valuable in challenging low-acceptance-rate regimes, where the proposed signal maintains its diagnostic utility. Collectively, these findings validate post-hoc signals as a valuable component for building more robust and intelligent LLM inference systems, and highlight a promising direction for future research on dynamic speculation length adaptation. |
| title | DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Information Theory I.2.7; C.2.4 |
| url | https://arxiv.org/abs/2509.01083 |