TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911411711508480 |
|---|---|
| author | Jiang, Haoyun He, Junqi Hong, Feng Yang, Xinlong Zhang, Jianwei Li, Zheng Zhuge, Zhengyang Chen, Zhiyong Han, Bo Lin, Junyang Yao, Jiangchao |
| author_facet | Jiang, Haoyun He, Junqi Hong, Feng Yang, Xinlong Zhang, Jianwei Li, Zheng Zhuge, Zhengyang Chen, Zhiyong Han, Bo Lin, Junyang Yao, Jiangchao |
| contents | Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35\% speedup over standard SD, with up to 50\% fewer target model invocations while maintaining comparable accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_23180 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification Jiang, Haoyun He, Junqi Hong, Feng Yang, Xinlong Zhang, Jianwei Li, Zheng Zhuge, Zhengyang Chen, Zhiyong Han, Bo Lin, Junyang Yao, Jiangchao Machine Learning Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35\% speedup over standard SD, with up to 50\% fewer target model invocations while maintaining comparable accuracy. |
| title | TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.23180 |