Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.09632 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908344975884288 |
|---|---|
| author | Lin, Chi-Heng Gao, Shangqian Smith, James Seale Patel, Abhishek Tuli, Shikhar Shen, Yilin Jin, Hongxia Hsu, Yen-Chang |
| author_facet | Lin, Chi-Heng Gao, Shangqian Smith, James Seale Patel, Abhishek Tuli, Shikhar Shen, Yilin Jin, Hongxia Hsu, Yen-Chang |
| contents | Large Language Models (LLMs) have reshaped the landscape of artificial intelligence by demonstrating exceptional performance across various tasks. However, substantial computational requirements make their deployment challenging on devices with limited resources. Recently, compression methods using low-rank matrix techniques have shown promise, yet these often lead to degraded accuracy or introduce significant overhead in parameters and inference latency. This paper introduces \textbf{Mo}dular \textbf{De}composition (MoDeGPT), a novel structured compression framework that does not need recovery fine-tuning while resolving the above drawbacks. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions via reconstructing the module-level outputs. MoDeGPT is developed based on a theoretical framework that utilizes three well-established matrix decomposition algorithms -- Nyström approximation, CR decomposition, and SVD -- and applies them to our redefined transformer modules. Our comprehensive experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods that rely on gradient information, and saves 98% of compute costs on compressing a 13B model. On \textsc{Llama}-2/3 and OPT models, MoDeGPT maintains 90-95% zero-shot performance with 25-30% compression rates. Moreover, the compression can be done on a single GPU within a few hours and increases the inference throughput by up to 46%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09632 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoDeGPT: Modular Decomposition for Large Language Model Compression Lin, Chi-Heng Gao, Shangqian Smith, James Seale Patel, Abhishek Tuli, Shikhar Shen, Yilin Jin, Hongxia Hsu, Yen-Chang Machine Learning Computation and Language 15A23 (Primary) I.2.7 Large Language Models (LLMs) have reshaped the landscape of artificial intelligence by demonstrating exceptional performance across various tasks. However, substantial computational requirements make their deployment challenging on devices with limited resources. Recently, compression methods using low-rank matrix techniques have shown promise, yet these often lead to degraded accuracy or introduce significant overhead in parameters and inference latency. This paper introduces \textbf{Mo}dular \textbf{De}composition (MoDeGPT), a novel structured compression framework that does not need recovery fine-tuning while resolving the above drawbacks. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions via reconstructing the module-level outputs. MoDeGPT is developed based on a theoretical framework that utilizes three well-established matrix decomposition algorithms -- Nyström approximation, CR decomposition, and SVD -- and applies them to our redefined transformer modules. Our comprehensive experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods that rely on gradient information, and saves 98% of compute costs on compressing a 13B model. On \textsc{Llama}-2/3 and OPT models, MoDeGPT maintains 90-95% zero-shot performance with 25-30% compression rates. Moreover, the compression can be done on a single GPU within a few hours and increases the inference throughput by up to 46%. |
| title | MoDeGPT: Modular Decomposition for Large Language Model Compression |
| topic | Machine Learning Computation and Language 15A23 (Primary) I.2.7 |
| url | https://arxiv.org/abs/2408.09632 |