Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910096902062080 |
|---|---|
| author | Qi, Ruoling Liu, Yirui Wu, Xuaner Wang, Xiangyu Li, Ming Chen, Chen Chen, Jian Chen, Yin Weng, Qizhen |
| author_facet | Qi, Ruoling Liu, Yirui Wu, Xuaner Wang, Xiangyu Li, Ming Chen, Chen Chen, Jian Chen, Yin Weng, Qizhen |
| contents | The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code will be released upon acceptance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_01609 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression Qi, Ruoling Liu, Yirui Wu, Xuaner Wang, Xiangyu Li, Ming Chen, Chen Chen, Jian Chen, Yin Weng, Qizhen Computation and Language The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code will be released upon acceptance. |
| title | Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.01609 |