Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917880265703424 |
|---|---|
| author | Bai, Guangji Chai, Zheng Ling, Chen Wang, Shiyu Lu, Jiaying Zhang, Nan Shi, Tingwei Yu, Ziyang Zhu, Mengdan Zhang, Yifei Song, Xinyuan Yang, Carl Cheng, Yue Zhao, Liang |
| author_facet | Bai, Guangji Chai, Zheng Ling, Chen Wang, Shiyu Lu, Jiaying Zhang, Nan Shi, Tingwei Yu, Ziyang Zhu, Mengdan Zhang, Yifei Song, Xinyuan Yang, Carl Cheng, Yue Zhao, Liang |
| contents | The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_00625 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models Bai, Guangji Chai, Zheng Ling, Chen Wang, Shiyu Lu, Jiaying Zhang, Nan Shi, Tingwei Yu, Ziyang Zhu, Mengdan Zhang, Yifei Song, Xinyuan Yang, Carl Cheng, Yue Zhao, Liang Machine Learning The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape. |
| title | Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2401.00625 |