DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913859832381440 |
|---|---|
| author | Deng, Hexuan Jiao, Wenxiang Liu, Xuebo Li, Jing Zhang, Min Tu, Zhaopeng |
| author_facet | Deng, Hexuan Jiao, Wenxiang Liu, Xuebo Li, Jing Zhang, Min Tu, Zhaopeng |
| contents | Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs. Structured pruning reduces model size and speeds up inference but often causes uneven degradation across domains, leading to biased performance. To address this, we propose DRPruning, a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. Experiments in monolingual and multilingual settings show that DRPruning surpasses similarly sized models in both pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. Further analysis demonstrates the robustness of DRPruning towards various domains and distribution shifts. Furthermore, DRPruning can determine optimal reference losses and data ratios automatically, suggesting potential for broader applications. Code and scripts are available at https://github.com/hexuandeng/DRPruning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_14055 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization Deng, Hexuan Jiao, Wenxiang Liu, Xuebo Li, Jing Zhang, Min Tu, Zhaopeng Computation and Language Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs. Structured pruning reduces model size and speeds up inference but often causes uneven degradation across domains, leading to biased performance. To address this, we propose DRPruning, a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. Experiments in monolingual and multilingual settings show that DRPruning surpasses similarly sized models in both pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. Further analysis demonstrates the robustness of DRPruning towards various domains and distribution shifts. Furthermore, DRPruning can determine optimal reference losses and data ratios automatically, suggesting potential for broader applications. Code and scripts are available at https://github.com/hexuandeng/DRPruning. |
| title | DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.14055 |