Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.23966 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911425209827328 |
|---|---|
| author | Tian, Jiayi Solgi, Ryan Lu, Jinming Yang, Yifan Li, Hai Zhang, Zheng |
| author_facet | Tian, Jiayi Solgi, Ryan Lu, Jinming Yang, Yifan Li, Hai Zhang, Zheng |
| contents | Large Language Models (LLMs) have enabled remarkable progress in natural language processing, yet their high computational and memory demands pose challenges for deployment in resource-constrained environments. Although recent low-rank decomposition methods offer a promising path for structural compression, they often suffer from accuracy degradation, expensive calibration procedures, and result in inefficient model architectures that hinder real-world inference speedups. In this paper, we propose FLAT-LLM, a fast and accurate, training-free structural compression method based on fine-grained low-rank transformations in the activation space. Specifically, we reduce the hidden dimension by transforming the weights using truncated eigenvectors computed via head-wise Principal Component Analysis, and employ a greedy budget redistribution strategy to adaptively allocate ranks across decoders. FLAT-LLM achieves efficient and effective weight compression without recovery fine-tuning, which could complete the calibration within a few minutes. Evaluated across 5 models and 11 datasets, FLAT-LLM outperforms structural pruning baselines in generalization and downstream performance, while delivering inference speedups over decomposition-based methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23966 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression Tian, Jiayi Solgi, Ryan Lu, Jinming Yang, Yifan Li, Hai Zhang, Zheng Computation and Language Large Language Models (LLMs) have enabled remarkable progress in natural language processing, yet their high computational and memory demands pose challenges for deployment in resource-constrained environments. Although recent low-rank decomposition methods offer a promising path for structural compression, they often suffer from accuracy degradation, expensive calibration procedures, and result in inefficient model architectures that hinder real-world inference speedups. In this paper, we propose FLAT-LLM, a fast and accurate, training-free structural compression method based on fine-grained low-rank transformations in the activation space. Specifically, we reduce the hidden dimension by transforming the weights using truncated eigenvectors computed via head-wise Principal Component Analysis, and employ a greedy budget redistribution strategy to adaptively allocate ranks across decoders. FLAT-LLM achieves efficient and effective weight compression without recovery fine-tuning, which could complete the calibration within a few minutes. Evaluated across 5 models and 11 datasets, FLAT-LLM outperforms structural pruning baselines in generalization and downstream performance, while delivering inference speedups over decomposition-based methods. |
| title | FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.23966 |