Computational Bottlenecks of Training Small-scale Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916500381630464 |
|---|---|
| author | Ashkboos, Saleh Mirzadeh, Iman Alizadeh, Keivan Sekhavat, Mohammad Hossein Nabi, Moin Farajtabar, Mehrdad Faghri, Fartash |
| author_facet | Ashkboos, Saleh Mirzadeh, Iman Alizadeh, Keivan Sekhavat, Mohammad Hossein Nabi, Moin Farajtabar, Mehrdad Faghri, Fartash |
| contents | While large language models (LLMs) dominate the AI landscape, Small-scale large Language Models (SLMs) are gaining attention due to cost and efficiency demands from consumers. However, there is limited research on the training behavior and computational requirements of SLMs. In this study, we explore the computational bottlenecks of training SLMs (up to 2B parameters) by examining the effects of various hyperparameters and configurations, including GPU type, batch size, model size, communication protocol, attention type, and the number of GPUs. We assess these factors on popular cloud services using metrics such as loss per dollar and tokens per second. Our findings aim to support the broader adoption and optimization of language model training for low-resource AI research institutes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19456 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Computational Bottlenecks of Training Small-scale Large Language Models Ashkboos, Saleh Mirzadeh, Iman Alizadeh, Keivan Sekhavat, Mohammad Hossein Nabi, Moin Farajtabar, Mehrdad Faghri, Fartash Machine Learning While large language models (LLMs) dominate the AI landscape, Small-scale large Language Models (SLMs) are gaining attention due to cost and efficiency demands from consumers. However, there is limited research on the training behavior and computational requirements of SLMs. In this study, we explore the computational bottlenecks of training SLMs (up to 2B parameters) by examining the effects of various hyperparameters and configurations, including GPU type, batch size, model size, communication protocol, attention type, and the number of GPUs. We assess these factors on popular cloud services using metrics such as loss per dollar and tokens per second. Our findings aim to support the broader adoption and optimization of language model training for low-resource AI research institutes. |
| title | Computational Bottlenecks of Training Small-scale Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.19456 |