Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929693457907712 |
|---|---|
| author | Wallace, Tom Ezzati-Jivan, Naser Ombuki-Berman, Beatrice |
| author_facet | Wallace, Tom Ezzati-Jivan, Naser Ombuki-Berman, Beatrice |
| contents | Advancements in Natural Language Processing are heavily reliant on the Transformer architecture, whose improvements come at substantial resource costs due to ever-growing model sizes. This study explores optimization techniques, including Quantization, Knowledge Distillation, and Pruning, focusing on energy and computational efficiency while retaining performance. Among standalone methods, 4-bit Quantization significantly reduces energy use with minimal accuracy loss. Hybrid approaches, like NVIDIA's Minitron approach combining KD and Structured Pruning, further demonstrate promising trade-offs between size reduction and accuracy retention. A novel optimization equation is introduced, offering a flexible framework for comparing various methods. Through the investigation of these compression methods, we provide valuable insights for developing more sustainable and efficient LLMs, shining a light on the often-ignored concern of energy efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00046 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models Wallace, Tom Ezzati-Jivan, Naser Ombuki-Berman, Beatrice Machine Learning Computation and Language 68T50 Advancements in Natural Language Processing are heavily reliant on the Transformer architecture, whose improvements come at substantial resource costs due to ever-growing model sizes. This study explores optimization techniques, including Quantization, Knowledge Distillation, and Pruning, focusing on energy and computational efficiency while retaining performance. Among standalone methods, 4-bit Quantization significantly reduces energy use with minimal accuracy loss. Hybrid approaches, like NVIDIA's Minitron approach combining KD and Structured Pruning, further demonstrate promising trade-offs between size reduction and accuracy retention. A novel optimization equation is introduced, offering a flexible framework for comparing various methods. Through the investigation of these compression methods, we provide valuable insights for developing more sustainable and efficient LLMs, shining a light on the often-ignored concern of energy efficiency. |
| title | Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models |
| topic | Machine Learning Computation and Language 68T50 |
| url | https://arxiv.org/abs/2502.00046 |