Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.08314 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913104191815680 |
|---|---|
| author | Wu, Wenhao Shao, Zishan Cui, Kangning Kim, Jinhee Wang, Yixiao Ye, Hancheng Zhuo, Danyang Chen, Yiran |
| author_facet | Wu, Wenhao Shao, Zishan Cui, Kangning Kim, Jinhee Wang, Yixiao Ye, Hancheng Zhuo, Danyang Chen, Yiran |
| contents | SVD-based Low-rank compression reduces transformer parameters and nominal FLOPs, but these savings often translate poorly into real LLM serving speedups. We show that this gap is largely a runtime problem: factorized checkpoints fragment execution paths, and the resulting overhead differs substantially between prefill and autoregressive decode. We present FlashSVD v1.5, a unified inference runtime for serving SVD-compressed transformers. FlashSVD v1.5 maps diverse public SVD compression families to a common factorized representation and combines phase-specific kernels with dense-KV decode, packed MLP execution, and per-layer CUDA-graph replay to reorganize the low-rank serving path into a thin runtime. Across representative decoder-serving settings, FlashSVD v1.5 achieves up to 2.55x decode and 2.39x end-to-end speedup, and it attains 1.48x average decode and 1.44x average end-to-end speedup across multiple popular SVD compression families. These results suggest that practical low-rank acceleration requires runtime co-design, not compression algorithms alone. Our code is available at: https://github.com/Zishan-Shao/FlashSVD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_08314 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast Wu, Wenhao Shao, Zishan Cui, Kangning Kim, Jinhee Wang, Yixiao Ye, Hancheng Zhuo, Danyang Chen, Yiran Machine Learning Artificial Intelligence Performance SVD-based Low-rank compression reduces transformer parameters and nominal FLOPs, but these savings often translate poorly into real LLM serving speedups. We show that this gap is largely a runtime problem: factorized checkpoints fragment execution paths, and the resulting overhead differs substantially between prefill and autoregressive decode. We present FlashSVD v1.5, a unified inference runtime for serving SVD-compressed transformers. FlashSVD v1.5 maps diverse public SVD compression families to a common factorized representation and combines phase-specific kernels with dense-KV decode, packed MLP execution, and per-layer CUDA-graph replay to reorganize the low-rank serving path into a thin runtime. Across representative decoder-serving settings, FlashSVD v1.5 achieves up to 2.55x decode and 2.39x end-to-end speedup, and it attains 1.48x average decode and 1.44x average end-to-end speedup across multiple popular SVD compression families. These results suggest that practical low-rank acceleration requires runtime co-design, not compression algorithms alone. Our code is available at: https://github.com/Zishan-Shao/FlashSVD. |
| title | FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast |
| topic | Machine Learning Artificial Intelligence Performance |
| url | https://arxiv.org/abs/2605.08314 |