Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.18567 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915874367078400 |
|---|---|
| author | Li, Shenggui Wang, Chao Zhu, Yikai Wang, Yubo Yin, Fan Shi, Shuai Chen, Yefei Dong, Xiaomin Chen, Qiaoling Pan, Jin Li, Ji Xie, Laixin Zhang, Yineng Yu, Lei Wen, Yonggang Tsang, Ivor Zhang, Tianwei |
| author_facet | Li, Shenggui Wang, Chao Zhu, Yikai Wang, Yubo Yin, Fan Shi, Shuai Chen, Yefei Dong, Xiaomin Chen, Qiaoling Pan, Jin Li, Ji Xie, Laixin Zhang, Yineng Yu, Lei Wen, Yonggang Tsang, Ivor Zhang, Tianwei |
| contents | Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_18567 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding Li, Shenggui Wang, Chao Zhu, Yikai Wang, Yubo Yin, Fan Shi, Shuai Chen, Yefei Dong, Xiaomin Chen, Qiaoling Pan, Jin Li, Ji Xie, Laixin Zhang, Yineng Yu, Lei Wen, Yonggang Tsang, Ivor Zhang, Tianwei Machine Learning Artificial Intelligence Computation and Language Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment. |
| title | SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2603.18567 |