Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.17946 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910679578968064 |
|---|---|
| author | Bai, Guangji Li, Yijiang Ling, Chen Kim, Kibaek Zhao, Liang |
| author_facet | Bai, Guangji Li, Yijiang Ling, Chen Kim, Kibaek Zhao, Liang |
| contents | The transformative impact of large language models (LLMs) like LLaMA and GPT on natural language processing is countered by their prohibitive computational demands. Pruning has emerged as a pivotal compression strategy, introducing sparsity to enhance both memory and computational efficiency. Yet, traditional global pruning is impractical for LLMs due to scalability issues, while local pruning, despite its efficiency, leads to suboptimal solutions. Addressing these challenges, we propose SparseLLM, a novel framework that redefines the global pruning process into manageable, coordinated subproblems, allowing for resource-efficient optimization with global optimality. SparseLLM's approach, which conceptualizes LLMs as a chain of modular functions and leverages auxiliary variables for problem decomposition, not only facilitates a pragmatic application on LLMs but also demonstrates significant performance improvements, particularly in high-sparsity regimes where it surpasses current state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17946 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SparseLLM: Towards Global Pruning for Pre-trained Language Models Bai, Guangji Li, Yijiang Ling, Chen Kim, Kibaek Zhao, Liang Computation and Language The transformative impact of large language models (LLMs) like LLaMA and GPT on natural language processing is countered by their prohibitive computational demands. Pruning has emerged as a pivotal compression strategy, introducing sparsity to enhance both memory and computational efficiency. Yet, traditional global pruning is impractical for LLMs due to scalability issues, while local pruning, despite its efficiency, leads to suboptimal solutions. Addressing these challenges, we propose SparseLLM, a novel framework that redefines the global pruning process into manageable, coordinated subproblems, allowing for resource-efficient optimization with global optimality. SparseLLM's approach, which conceptualizes LLMs as a chain of modular functions and leverages auxiliary variables for problem decomposition, not only facilitates a pragmatic application on LLMs but also demonstrates significant performance improvements, particularly in high-sparsity regimes where it surpasses current state-of-the-art methods. |
| title | SparseLLM: Towards Global Pruning for Pre-trained Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.17946 |