Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913705095069696 |
|---|---|
| author | Islam, Ashhadul Belhaouari, Samir Brahim Bermak, Amine |
| author_facet | Islam, Ashhadul Belhaouari, Samir Brahim Bermak, Amine |
| contents | The exponential growth of large language models (LLMs) like ChatGPT has revolutionized artificial intelligence, offering unprecedented capabilities in natural language processing. However, the extensive computational resources required for training these models have significant environmental implications, including high carbon emissions, energy consumption, and water usage. This research presents a novel approach to LLM pruning, focusing on the systematic evaluation of individual weight importance throughout the training process. By monitoring parameter evolution over time, we propose a method that effectively reduces model size without compromising performance. Extensive experiments with both a scaled-down LLM and a large multimodal model reveal that moderate pruning enhances efficiency and reduces loss, while excessive pruning drastically deteriorates model performance. These findings highlight the critical need for optimized AI models to ensure sustainable development, balancing technological advancement with environmental responsibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17071 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability Islam, Ashhadul Belhaouari, Samir Brahim Bermak, Amine Computation and Language Artificial Intelligence The exponential growth of large language models (LLMs) like ChatGPT has revolutionized artificial intelligence, offering unprecedented capabilities in natural language processing. However, the extensive computational resources required for training these models have significant environmental implications, including high carbon emissions, energy consumption, and water usage. This research presents a novel approach to LLM pruning, focusing on the systematic evaluation of individual weight importance throughout the training process. By monitoring parameter evolution over time, we propose a method that effectively reduces model size without compromising performance. Extensive experiments with both a scaled-down LLM and a large multimodal model reveal that moderate pruning enhances efficiency and reduces loss, while excessive pruning drastically deteriorates model performance. These findings highlight the critical need for optimized AI models to ensure sustainable development, balancing technological advancement with environmental responsibility. |
| title | Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.17071 |