Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912795127185408 |
|---|---|
| author | Guan, Yue Yu, Changming Fang, Shihan Hu, Weiming Pan, Zaifeng Wang, Zheng Liu, Zihan Zhou, Yangjie Ding, Yufei Guo, Minyi Leng, Jingwen |
| author_facet | Guan, Yue Yu, Changming Fang, Shihan Hu, Weiming Pan, Zaifeng Wang, Zheng Liu, Zihan Zhou, Yangjie Ding, Yufei Guo, Minyi Leng, Jingwen |
| contents | Speculative decoding improves LLM inference by generating and verifying multiple tokens in parallel, but existing systems suffer from suboptimal performance due to a mismatch between dynamic speculation and static runtime assumptions. We present Yggdrasil, a co-designed system that enables latency-optimal speculative decoding through context-aware tree drafting and compiler-friendly execution. Yggdrasil introduces an equal-growth tree structure for static graph compatibility, a latency-aware optimization objective for draft selection, and stage-based scheduling to reduce overhead. Yggdrasil supports unmodified LLMs and achieves up to $3.98\times$ speedup over state-of-the-art baselines across multiple hardware setups. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_23858 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding Guan, Yue Yu, Changming Fang, Shihan Hu, Weiming Pan, Zaifeng Wang, Zheng Liu, Zihan Zhou, Yangjie Ding, Yufei Guo, Minyi Leng, Jingwen Machine Learning Programming Languages Speculative decoding improves LLM inference by generating and verifying multiple tokens in parallel, but existing systems suffer from suboptimal performance due to a mismatch between dynamic speculation and static runtime assumptions. We present Yggdrasil, a co-designed system that enables latency-optimal speculative decoding through context-aware tree drafting and compiler-friendly execution. Yggdrasil introduces an equal-growth tree structure for static graph compatibility, a latency-aware optimization objective for draft selection, and stage-based scheduling to reduce overhead. Yggdrasil supports unmodified LLMs and achieves up to $3.98\times$ speedup over state-of-the-art baselines across multiple hardware setups. |
| title | Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding |
| topic | Machine Learning Programming Languages |
| url | https://arxiv.org/abs/2512.23858 |