Reducing the Barriers to Entry for Foundation Model Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912070759350272 |
|---|---|
| author | Faraboschi, Paolo Giles, Ellis Hotard, Justin Owczarek, Konstanty Wheeler, Andrew |
| author_facet | Faraboschi, Paolo Giles, Ellis Hotard, Justin Owczarek, Konstanty Wheeler, Andrew |
| contents | The world has recently witnessed an unprecedented acceleration in demands for Machine Learning and Artificial Intelligence applications. This spike in demand has imposed tremendous strain on the underlying technology stack in supply chain, GPU-accelerated hardware, software, datacenter power density, and energy consumption. If left on the current technological trajectory, future demands show insurmountable spending trends, further limiting market players, stifling innovation, and widening the technology gap. To address these challenges, we propose a fundamental change in the AI training infrastructure throughout the technology ecosystem. The changes require advancements in supercomputing and novel AI training approaches, from high-end software to low-level hardware, microprocessor, and chip design, while advancing the energy efficiency required by a sustainable infrastructure. This paper presents the analytical framework that quantitatively highlights the challenges and points to the opportunities to reduce the barriers to entry for training large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_08811 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reducing the Barriers to Entry for Foundation Model Training Faraboschi, Paolo Giles, Ellis Hotard, Justin Owczarek, Konstanty Wheeler, Andrew Emerging Technologies Artificial Intelligence Hardware Architecture Machine Learning The world has recently witnessed an unprecedented acceleration in demands for Machine Learning and Artificial Intelligence applications. This spike in demand has imposed tremendous strain on the underlying technology stack in supply chain, GPU-accelerated hardware, software, datacenter power density, and energy consumption. If left on the current technological trajectory, future demands show insurmountable spending trends, further limiting market players, stifling innovation, and widening the technology gap. To address these challenges, we propose a fundamental change in the AI training infrastructure throughout the technology ecosystem. The changes require advancements in supercomputing and novel AI training approaches, from high-end software to low-level hardware, microprocessor, and chip design, while advancing the energy efficiency required by a sustainable infrastructure. This paper presents the analytical framework that quantitatively highlights the challenges and points to the opportunities to reduce the barriers to entry for training large language models. |
| title | Reducing the Barriers to Entry for Foundation Model Training |
| topic | Emerging Technologies Artificial Intelligence Hardware Architecture Machine Learning |
| url | https://arxiv.org/abs/2404.08811 |