ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912445463789568 |
|---|---|
| author | Liu, Hongyi Saha, Rajarshi Jia, Zhen Park, Youngsuk Huang, Jiaji Sabach, Shoham Wang, Yu-Xiang Karypis, George |
| author_facet | Liu, Hongyi Saha, Rajarshi Jia, Zhen Park, Youngsuk Huang, Jiaji Sabach, Shoham Wang, Yu-Xiang Karypis, George |
| contents | Large Language Models (LLMs) have demonstrated exceptional performance in natural language processing tasks, yet their massive size makes serving them inefficient and costly. Semi-structured pruning has emerged as an effective method for model acceleration, but existing approaches are suboptimal because they focus on local, layer-wise optimizations using heuristic rules, failing to leverage global feedback. We present ProxSparse, a learning-based framework for mask selection enabled by regularized optimization. ProxSparse transforms the rigid, non-differentiable mask selection process into a smoother optimization procedure, allowing gradual mask exploration with flexibility. ProxSparse does not involve additional weight updates once the mask is determined. Our extensive evaluations on 7 widely used models show that ProxSparse consistently outperforms previously proposed semi-structured mask selection methods with significant improvement, demonstrating the effectiveness of our learned approach towards semi-structured pruning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00258 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs Liu, Hongyi Saha, Rajarshi Jia, Zhen Park, Youngsuk Huang, Jiaji Sabach, Shoham Wang, Yu-Xiang Karypis, George Machine Learning Computation and Language Large Language Models (LLMs) have demonstrated exceptional performance in natural language processing tasks, yet their massive size makes serving them inefficient and costly. Semi-structured pruning has emerged as an effective method for model acceleration, but existing approaches are suboptimal because they focus on local, layer-wise optimizations using heuristic rules, failing to leverage global feedback. We present ProxSparse, a learning-based framework for mask selection enabled by regularized optimization. ProxSparse transforms the rigid, non-differentiable mask selection process into a smoother optimization procedure, allowing gradual mask exploration with flexibility. ProxSparse does not involve additional weight updates once the mask is determined. Our extensive evaluations on 7 widely used models show that ProxSparse consistently outperforms previously proposed semi-structured mask selection methods with significant improvement, demonstrating the effectiveness of our learned approach towards semi-structured pruning. |
| title | ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.00258 |