A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914969111494656 |
|---|---|
| author | Guo, Cong Cheng, Feng Du, Zhixu Kiessling, James Ku, Jonathan Li, Shiyu Li, Ziru Ma, Mingyuan Molom-Ochir, Tergel Morris, Benjamin Shan, Haoxuan Sun, Jingwei Wang, Yitu Wei, Chiyue Wu, Xueying Wu, Yuhao Yang, Hao Frank Zhang, Jingyang Zhang, Junyao Zheng, Qilin Zhou, Guanglei Hai Li Chen, Yiran |
| author_facet | Guo, Cong Cheng, Feng Du, Zhixu Kiessling, James Ku, Jonathan Li, Shiyu Li, Ziru Ma, Mingyuan Molom-Ochir, Tergel Morris, Benjamin Shan, Haoxuan Sun, Jingwei Wang, Yitu Wei, Chiyue Wu, Xueying Wu, Yuhao Yang, Hao Frank Zhang, Jingyang Zhang, Junyao Zheng, Qilin Zhou, Guanglei Hai Li Chen, Yiran |
| contents | The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07265 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Guo, Cong Cheng, Feng Du, Zhixu Kiessling, James Ku, Jonathan Li, Shiyu Li, Ziru Ma, Mingyuan Molom-Ochir, Tergel Morris, Benjamin Shan, Haoxuan Sun, Jingwei Wang, Yitu Wei, Chiyue Wu, Xueying Wu, Yuhao Yang, Hao Frank Zhang, Jingyang Zhang, Junyao Zheng, Qilin Zhou, Guanglei Hai Li Chen, Yiran Hardware Architecture Artificial Intelligence Machine Learning Software Engineering The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems. |
| title | A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models |
| topic | Hardware Architecture Artificial Intelligence Machine Learning Software Engineering |
| url | https://arxiv.org/abs/2410.07265 |